How to Scrape GitHub Data: Complete Guide for 2026
This guide covers extracting publicly accessible data. Always review a site's robots.txt and Terms of Service before scraping.
TL;DR
Scrape public GitHub profiles, repositories, and code using AlterLab's API with automatic tier escalation. Use Python or Node.js SDKs to extract structured data from pages like github.com/user or github.com/user/repo while respecting rate limits and avoiding anti-bot blocks. Start at T1 and let the API promote to T3/T4 as needed for JS-rendered content.
Why collect developer data from GitHub?
GitHub hosts the largest public dataset of developer activity, making it invaluable for:
- Talent sourcing: Identify engineers with specific language expertise by analyzing public repo contributions and commit history for recruitment pipelines
- Market intelligence: Monitor technology adoption trends by tracking star/fork growth on framework repositories to inform product roadmap decisions
- Security research: Scan public code for accidental credential leaks or vulnerable dependencies across thousands of repositories at scale
Technical challenges
GitHub presents two primary obstacles for scrapers:
- Aggressive rate limiting: Unauthenticated IP addresses face low request thresholds (often 10-20/minute) before receiving 403 responses
- Dynamic content rendering: Key data like contribution graphs, language statistics, and repo metadata load via JavaScript after initial HTML
Raw HTTP requests fail because GitHub's server delivers minimal HTML skeletons requiring client-side JS execution. AlterLab's Smart Rendering API handles this by automatically promoting to browser-based tiers (T3/T4) when static retrieval fails, executing JavaScript in headless Chrome to capture fully rendered DOM while managing proxy rotation and fingerprint evasion.
Quick start with AlterLab API
Begin by installing the SDK and making your first request. See the Getting started guide for detailed setup.
Here's how to scrape a public GitHub profile in Python:
```python title="scrape_github-com.py" {3-5}
client = alterlab.Client("YOUR_API_KEY")
response = client.scrape("https://github.com/microsoft")
print(response.text) # Contains rendered HTML with contribution graph
Equivalent Node.js implementation:
```javascript title="scrape_github-com.js" {3-5}
const client = new AlterLab({ apiKey: "YOUR_API_KEY" });
const response = await client.scrape("https://github.com/microsoft");
console.log(response.text);
And the cURL equivalent for quick testing:
```bash title="Terminal"
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-d '{"url": "https://github.com/microsoft"}'
<div data-infographic="steps">
<div data-step data-number="1" data-title="Request sent" data-description="API receives URL and initiates scrape at T1 (static HTML)"></div>
<div data-step data-number="2" data-title="Tier escalation" data-description="If JS detection fails, promotes to T3 (stealth) or T4 (browser) automatically"></div>
<div data-step data-number="3" data-style="success" data-title="Data returned" data-description="Fully rendered HTML or structured JSON delivered to your application"></div>
</div>
## Extracting structured data
Target these CSS selectors for common GitHub profile data points:
- Contribution count: `svg[data-test-id="contribution-graph-labels"] text`
- Repository list: `repo-list-item h3 a` (extract `href` for repo URLs)
- Language stats: `li[data-hovercard-type="language"] span.Stats"]` (percentage values)
- Follower count: `a[href*="?tab=followers"] span.text-bold`
Example extracting follower count with Python:
```python
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, 'html.parser')
followers = soup.select_one('a[href*="?tab=followers"] span.text-bold').text
print(f"Followers: {followers}")
Structured JSON extraction with Cortex
For typed data without parsing HTML, use AlterLab's Cortex AI extraction. Define a schema for repo metadata:
```python title="extract_github-com_structured.py"
client = alterlab.Client("YOUR_API_KEY")
result = client.extract(
url="https://github.com/microsoft/vscode",
schema={
"type": "object",
"properties": {
"repoName": {"type": "string"},
"description": {"type": "string"},
"stars": {"type": "integer"},
"forks": {"type": "integer"},
"language": {"type": "string"},
"license": {"type": "string"}
},
"required": ["repoName", "stars"]
}
)
print(result.data) # Output: {"repoName": "vscode", "stars": 150000, ...}
Cortex handles JavaScript rendering and element selection automatically, returning validated JSON matching your schema. This eliminates brittle CSS selector maintenance when GitHub updates its UI.
## Cost breakdown
GitHub's dynamic content typically requires T3 (Stealth) tier for reliable access. AlterLab's pricing scales with complexity:
| Tier | Use Case | Cost per Request | Cost per 1,000 | Requests per $1 |
|------|----------|-----------------|----------------|------------------|
| T1 — Curl | Static HTML, no JS needed | $0.0002 | $0.20 | 5,000 |
| T2 — HTTP | Standard pages with headers | $0.0003 | $0.30 | 3,333 |
| T3 — Stealth | Protected pages, anti-bot active | $0.002 | $2.00 | 500 |
| T4 — Browser | Full JS rendering required | $0.004 | $4.00 | 250 |
| T5 — CAPTCHA | CAPTCHA solving + JS rendering | $0.02 | $20.00 | 50 |
[AlterLab pricing](/pricing) shows volume discounts starting at 100,000 monthly requests. Note: AlterLab auto-escalates tiers — start at T1 and the API promotes automatically if a lower tier fails. You only pay for the tier that succeeds. For most GitHub profiles and repos, T3 suffices at $0.002/request.
## Best practices
Follow these guidelines for responsible GitHub scraping:
- **Rate limiting**: Implement exponential backoff (start at 1s delay, double after 429) staying under 10 requests/minute per IP for unauthenticated scraping
- **Robots.txt compliance**: GitHub's `/robots.txt` disallows scraping `/private/*` and `/security/*` paths — restrict to public paths like `/[^/]+/[^/]+/*`
- **Request headers**: Include a realistic `User-Agent` (e.g., `Mozilla/5.0 (compatible; MyScraper/1.0; +https://yoursite.com/bot)`) and accept `text/html`
- **Error handling**: Retry 429/503 responses 3x with jitter; treat 404 as permanent failure for that URL
- **Data freshness**: For frequently changing data (like contribution graphs), cache results for 1-4 hours based on update frequency
## Scaling up
For large-scale GitHub data collection:
- **Batch processing**: Use AlterLab's batch endpoint (`/v1/scrape/batch`) to send 100 URLs per request, reducing overhead
- **Scheduling**: Trigger daily scrapes via cron or AlterLab's built-in scheduler for trending analysis
- **Handling large datasets**: Stream results directly to your data warehouse using webhooks instead of storing intermediate HTML
- **Team collaboration**: Share API keys within your [Organizations](/docs/teams) with role-based access and unified billing
## Key takeaways
- GitHub scraping requires JS rendering and rate limit management — AlterLab handles both through automatic tier escalation
- Focus on public data only: profiles, public repositories, and open-source code statistics
- Start with T1 requests; the API promotes to T3/T4 as needed for dynamic content at no extra failed request cost
- Use Cortex for schema-validated JSON output to bypass fragile HTML parsing
- Always implement polite rate limiting and respect robots.txt regardless of tool capabilities
[Explore more GitHub scraping patterns](/scrape/github) in our dedicated guide.
Top comments (0)