Google Scholar is the single most important search engine for academic research. Billions of papers indexed, citation counts, author profiles, related work links. And Google has never released an official API for it.
Not deprecated. Not restricted. Just... never built one.
If you want to programmatically search Google Scholar, grab paper titles, authors, citation counts, and PDF links, you are on your own. So I built an actor that does exactly that.
What It Pulls
You give it a search query (like "transformer architecture attention mechanism") and it returns structured data:
{
"title": "Attention Is All You Need",
"authors": "A Vaswani, N Shazeer, N Parmar...",
"citationCount": 112847,
"year": "2017",
"url": "https://arxiv.org/abs/1706.03762",
"pdfUrl": "https://arxiv.org/pdf/1706.03762",
"snippet": "The dominant sequence transduction models are based on complex recurrent..."
}
Paper titles, author lists, citation counts, publication year, direct links, and PDF URLs when available. Everything a researcher needs to build a literature review or track citations over time.
The Numbers Tell a Story
Here's where it gets interesting. The actor has 22 users and 5,065 total runs. Do the math on that ratio: 230 runs per user on average.
These are not casual users clicking "Run" once to test it. These are power users running it at scale. Academics building citation databases. Research firms tracking publication trends across thousands of queries. AI companies monitoring new papers in their domain.
That run to user ratio is the strongest signal I have that this tool solves a real problem. When someone runs your tool 200+ times, they have built it into a workflow.
Why Scholar Is Hard to Scrape
Google Scholar is notoriously aggressive about blocking automated access. It will throw CAPTCHAs after just a handful of requests from the same IP. Most simple scraping scripts break within minutes.
The actor handles this with:
- Proxy rotation across residential IPs
- Session management to maintain cookies between requests
- Randomized delays that mimic human browsing patterns
- Automatic retry logic when a request gets blocked
I also had to deal with Google's inconsistent HTML. Scholar's markup changes subtly over time. Element class names shift, layout structures get tweaked. The parser needs regular maintenance to keep working.
Who Uses This
Three main groups keep showing up:
Academics and PhD students building systematic literature reviews. Instead of manually searching and copying results, they run batch queries and get structured data they can feed into reference managers or spreadsheets.
Research firms and think tanks tracking publication trends. They want to know how many papers mention "large language models" per quarter, or which authors are publishing most frequently in a specific subfield.
AI and ML teams monitoring state of the art. When a new paper drops with high early citation velocity, they want to know about it fast.
Try It
The actor is on the Apify Store with pay per result pricing ($0.004 per paper): Google Scholar Scraper
If you have ever copy pasted results from Google Scholar into a spreadsheet, this will save you hours. And if you are doing it at scale, it will save you from getting IP banned.
Built in Nairobi by George. 40+ actors, 5,000+ runs on Scholar alone.
Top comments (0)