DEV Community

Alex Spinov
Alex Spinov

Posted on

Free Web Scraping: Run Scrapers on GitHub Actions (No Server, No Cost)

#webdev #github #python #tutorial

I used to pay $5/month for a server to run my scrapers. Then I realized GitHub Actions gives me 2,000 free minutes per month.

That's enough to run a scraper every hour, 24/7, for $0.

The Setup (5 Minutes)

You need:

  1. A GitHub repo
  2. A Python script
  3. A YAML config file

That's it.

Step 1: The Scraper 

# scraper.py
import requests
import json
from datetime import datetime
import os

def scrape():
    # Example: track Bitcoin price
    data = requests.get(
        "https://api.coingecko.com/api/v3/simple/price",
        params={"ids": "bitcoin,ethereum", "vs_currencies": "usd"}
    ).json()

    entry = {
        "timestamp": datetime.utcnow().isoformat(),
        "bitcoin": data["bitcoin"]["usd"],
        "ethereum": data["ethereum"]["usd"],
    }

    os.makedirs("data", exist_ok=True)
    try:
        with open("data/prices.json") as f:
            history = json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        history = []

    history.append(entry)
    with open("data/prices.json", "w") as f:
        json.dump(history, f, indent=2)

    print(f"BTC: ${entry['bitcoin']:,} | ETH: ${entry['ethereum']:,}")

if __name__ == "__main__":
    scrape()
Enter fullscreen mode Exit fullscreen mode

Step 2: The GitHub Action

Create .github/workflows/scrape.yml:

name: Scrape Data
on:
  schedule:
    - cron: '0 * * * *'  # Every hour
  workflow_dispatch:       # Manual trigger button

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install requests
      - run: python scraper.py
      - name: Commit data
        run: |
          git config user.name 'GitHub Action'
          git config user.email 'action@github.com'
          git add -A
          git diff --staged --quiet || git commit -m 'Update $(date -u +%Y-%m-%dT%H:%M)'
          git push
Enter fullscreen mode Exit fullscreen mode

Step 3: Push and Forget 

git add -A && git commit -m 'Add scraper' && git push
Enter fullscreen mode Exit fullscreen mode

Go to your repo → Actions tab → you'll see it running.

What Can You Scrape?

Anything with a free API:

  • Crypto prices (CoinGecko — no auth needed)
  • Stock data (Alpha Vantage — free key)
  • Weather (Open-Meteo — no auth)
  • HN front page (Firebase API — no auth)
  • Earthquake data (USGS — no auth)
  • Exchange rates (Open Exchange Rates — no auth)

Scheduling Cheat Sheet

Schedule Cron
Every hour 0 * * * *
Every 6 hours 0 */6 * * *
Daily midnight 0 0 * * *
Weekly Monday 0 0 * * 1

Why Not a Real Server?

GitHub Actions $5 VPS
Cost $0 $60/year
Setup time 5 min 30+ min
Maintenance Zero Updates, security
Data backup Git history You manage
Uptime 99.9% You monitor

Limitations

  • Max 2,000 min/month on free tier (plenty for hourly scrapes)
  • Cron accuracy: +/- 15 minutes (GitHub queues jobs)
  • No persistent state between runs (use git commits)
  • Max 6 hours per job

Template Repo

Fork this to get started: github-action-scraper-template

More data tools:

What data would you track with a free hourly scraper?

Top comments (0)