Truffle Pig Data

Posted on Jul 25

Google Scholar API: How to Scrape Papers, Citations, and Author Profiles in 2026

#webscraping #python #api #datascience

Anyone who has run a literature review knows the drill: forty browser tabs on Google Scholar, citation counts copied into a spreadsheet, and a captcha every few minutes for your trouble. I built my last review dataset the painful way before switching tools. This post shows the manual route and where it breaks, then the shortcut: the Google Scholar API on Apify, one Actor with six modes covering papers, citations, author profiles, and co-author networks.

Disclosure: the Apify links in this post are affiliate links. If you run the Actor, I may earn a referral commission at no extra cost to you.

Does Google Scholar have an API?

No, and it never has. There is no official developer endpoint for Scholar, no key to request, and the site is famously quick to throw captchas at anything that smells automated. So a Google Scholar API in practice means a scraper you consume like an API: pass a mode and a query, get structured JSON back, with pagination and parsing handled for you.

What the Google Scholar API returns

The Google Scholar API returns research papers, citation formats, author profiles with h-index and i10-index, publication lists, per-article citation history, and co-author networks as structured JSON.

Field	Example	Notes
`paper_title`	`"Attention Is All You Need"`	Search mode, one row per paper
`publication_info.summary`	`"A Vaswani, N Shazeer, N Parmar - Advances in NIPS, 2017"`	Authors, venue, year
`inline_links.cited_by_total`	`120000`	Citation count per paper
`result_id`	`"K7uerNYAAAAJ:u5HHmVD_uO8C"`	Feeds the `cite` mode for BibTeX and APA
`cited_by_summary.h_index_all`	`150`	Author profile mode
`cited_by_graph`	`[{"year": 2019, "citations": 45000}]`	Year-by-year citation history

The mode input picks the operation: search, cite, author_profile, author_articles, author_citation, or author_co_authors. Search first; each result carries the result_id and author_id values the other modes need.

Who this is for

Grad students and researchers assembling literature reviews, bibliometrics people tracking h-index and citation growth over time, and developers feeding academic data into agents or RAG pipelines.

The manual way, and where it breaks

The DIY version scrapes scholar.google.com with requests and a parser. It dies fast. Scholar rate-limits harder than almost any Google property, so captchas arrive within pages. Author profiles paginate through button clicks, not URLs. Citation exports hide behind a popup per paper. Multiply that by a few hundred papers and you are running a captcha farm, not a research project. I gave up on my own version the second time the parser broke in a month.

The faster way: run the Google Scholar scraper

Apify Console

Open the Google Scholar API and click Try for free.
Pick a mode and fill the required fields (q for search, author_id for author modes).
Run it and download the dataset as JSON, CSV, or Excel.

REST

curl -X POST "https://api.apify.com/v2/acts/johnvc~google-scholar-api/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "mode": "search", "q": "transformer neural network", "as_ylo": 2020, "as_yhi": 2024, "num": 10, "max_pages": 3 }'

Run endpoint reference: the Apify API docs.

Search Google Scholar in Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("johnvc/google-scholar-api").call(
    run_input={
        "mode": "search",
        "q": "retrieval augmented generation",
        "as_ylo": 2022,
        "max_pages": 2,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    cites = (item.get("inline_links") or {}).get("cited_by_total")
    print(item["paper_title"], cites, item.get("link"))

The task Google Scholar API: papers, authors and citations is the same starting point with the input prefilled.

Find every paper citing a landmark work

Reverse citation search is where Scholar shines and where scraping it by hand hurts most. The task Find papers citing Attention Is All You Need uses the cites parameter to pull everything citing one landmark paper, with authors, year, and citation counts per row.

Get citation counts for an author's whole bibliography

Get citation counts for every paper by an author runs author_articles with pagination, up to 100 papers per page, which is the raw material for any bibliometric analysis.

Export results to CSV

Export Google Scholar results to CSV lands the same data as a spreadsheet, which is usually what a supervisor or co-author actually wants to receive.

Use it from Claude and other MCP clients

Add the Apify MCP server (https://mcp.apify.com/?tools=actors,docs,johnvc/google-scholar-api) and Claude, Claude Code, or Cursor picks the right mode from a plain question like "find the most-cited RAG papers since 2022". The task Search Google Scholar from Claude via MCP shows the setup, and claude.ai is where to grab Claude if you have not already.

The example repo

johnisanerd / Apify-Google-Scholar-API

Google Scholar API on Apify - Python (uv) + MCP example. Academic papers, citations, and author profiles (six modes) as structured JSON.

🎓 Google Scholar API: Papers, Citations, and Author Profiles in Clean JSON

The efficient, reliable, and developer-friendly way to use the Google Scholar API.

Actor page: apify.com/johnvc/google-scholar-api Input schema: apify.com/johnvc/google-scholar-api/input-schema

The Google Scholar API returns academic search results, citation formats, and author profiles as clean, structured JSON. A single mode parameter dispatches to six endpoints: search papers, get citation formats for a paper, or pull an author's profile, articles, citations, and co-authors. Each search result carries the title, authors, publication summary, cited-by count, versions, and links. Built for literature reviews, citation analysis, research dashboards, and AI agent workflows.

Video Walkthrough

Quick Start

Prerequisites

Python 3.11 or higher
An Apify account and API key (get a free key here)

Clone the repository

git clone https://github.com/johnisanerd/Apify-Google-Scholar-API.git
cd Apify-Google-Scholar-API

Install dependencies with UV

# Install UV if you do not have it:
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install project dependencies:

…

View on GitHub

The repo has a Python quick start plus MCP install steps for Claude, Cursor, and ChatGPT.

FAQ about scraping Google Scholar

How much does the Google Scholar scraper cost?

Pay per event, no subscription: a $0.02 setup fee per run plus $0.02 per query executed. Paginated modes count one query per page, single-shot modes count one per run. Worked examples: a 5-page search is $0.12, an author profile is $0.04. New Apify accounts include free platform credit.

Can the scraper export BibTeX and APA citations?

Yes. Run mode=cite with a result_id from a search and you get MLA, APA, Chicago, Harvard, and Vancouver strings plus BibTeX, EndNote, RefMan, and RefWorks export links. That replaces the click-per-paper citation popup.

Can Claude use the Google Scholar scraper through MCP?

Yes. Connected through the Apify MCP server, the Actor becomes a tool the agent can call, and it will choose between search and the author modes based on how you phrase the question.

Can I schedule the scraper for citation tracking?

Yes, and that is the best use of it. Save one task per topic or researcher, attach an Apify Schedule (weekly 0 9 * * 1 works well for citation snapshots), and store cited_by_graph over time. Start from the Google Scholar API.

Why did the scraper return fewer pages than max_pages?

Scholar stops when it runs out of matches, so pagination ends early rather than padding results. max_pages is a ceiling and a cost cap, not a promise.

Is this scraper enough for a full bibliometric analysis?

It supplies the inputs: publication lists, per-paper citation counts, h-index and i10-index, citation history, and co-author networks. The analysis itself, normalizing names and computing trends, still happens in your notebook.

More from Truffle Pig Data

Related Actors for a research pipeline: the Google Scholar Lite API for cheaper high-volume paper search, the Google Scholar Case Law API for US court opinions, and the Google Patents API for prior art.

Wrapping up

There is no official Google Scholar API, but six modes of structured JSON cover most of what a research workflow needs. Try the Google Scholar API, or clone the example repo and start from working code.

DEV Community