DEV Community

Cover image for I Built a Wikipedia Scraper That Runs on Apify Without Proxies
Prime Sieve
Prime Sieve

Posted on

I Built a Wikipedia Scraper That Runs on Apify Without Proxies

#scraping #apify #wikipedia #buildinpublic

I Built a Wikipedia Scraper That Runs on Apify Without Proxies

Most Wikipedia scrapers on Apify use Playwright or headless browsers. I built one that hits the MediaWiki API directly — 200 responses, zero proxy costs, 54 lines of code.

Why Wikipedia Data?

Wikipedia is the largest open knowledge base on the internet. 60M+ articles across 300+ languages. Researchers, NLP teams, knowledge graph builders, and content aggregators all need structured access to it.

The problem: most tools overcomplicate it. They spin up headless browsers to render pages, manage proxy pools to avoid rate limits, and charge you for the overhead.

What This Actor Does

Search any keyword across any Wikipedia language. Get back structured results: title, page ID, word count, snippet, timestamp, and direct URL. Paginate up to 500 results per keyword with automatic offset handling.

Input:

  • keywords — array of search terms
  • lang — language code (default: en)
  • maxResults — up to 500 per keyword

Output per result:

{
  "keyword": "quantum computing",
  "lang": "en",
  "title": "Quantum computing",
  "pageid": 22967,
  "size": 85432,
  "wordcount": 12400,
  "snippet": "Quantum computing is a type of computation...",
  "timestamp": "2024-12-15T10:30:00Z",
  "url": "https://en.wikipedia.org/wiki/Quantum_computing"
}
Enter fullscreen mode Exit fullscreen mode

The No-Proxy Advantage

Wikipedia's MediaWiki API is open. No authentication required. No rate limit tricks needed if you respect their guidelines.

This means:

  • Zero proxy costs — runs clean on Apify's AWS infrastructure
  • No browser overhead — plain HTTP GET, ~250ms per request
  • Reliable — no CAPTCHA challenges, no IP blocks, no flaky browser sessions

Most competing actors charge $3-5 per 1k results and burn proxy credits. This one runs at the Apify free tier.

Use Cases

  • NLP training data — bulk collect article metadata for text classification
  • Knowledge graph construction — page IDs + titles + timestamps for entity linking
  • Content gap analysis — compare coverage across languages
  • Research automation — systematic literature discovery on any topic
  • SEO research — find what Wikipedia covers about your niche

Running It

Search for "Wikipedia Search Scraper" on Apify Store, or use the API directly:

curl -X POST "https://api.apify.com/v2/acts/primesievecoder~wikipedia-search-scraper/runs" \
  -H "Content-Type: application/json" \
  -d '{"keywords": ["quantum computing", "machine learning"], "lang": "en", "maxResults": 100}'
Enter fullscreen mode Exit fullscreen mode

What's Next

Adding full-text extract mode (prop=extracts) for teams that need article body content, not just search metadata. Stay tuned.

Built by Prime Sieve — scraping tools that just work. No proxies, no headless browsers, no drama.

Top comments (0)