DEV Community

Felixwang007
Felixwang007

Posted on

I Built a Zero-Cost, Zero-API GitHub Trending Aggregator That Updates Itself Every 6 Hours

#ai #python #opensource #stocks

Every morning, I used to open github.com/trending, scroll through 5 pages, and lose 20 minutes before writing a single line of code. The worst part: there is no official GitHub Trending API, and every third-party wrapper either requires a token, rate-limits you after 10 requests, or charges money for what is essentially a public HTML page.

So I built my own aggregator. It costs $0 per month, requires zero API keys, and updates itself every 6 hours. The whole thing is ~200 lines of Python + one GitHub Actions workflow. Here's how it works.

The core trick: parse HTML, not JSON

GitHub renders Trending as server-side HTML, which means you can extract everything you need with a few regexes — no headless browser, no API token, no authentication.

import re, json, urllib.request

URL = "https://github.com/trending?since=daily"

html = urllib.request.urlopen(URL).read().decode("utf-8")

# Each trending repo is an <article> block
articles = re.findall(r'<article[^>]*>(.*?)</article>', html, re.S)

repos = []
for a in articles:
    name = re.search(r'href="/([^"]+)"', a).group(1)
    desc = re.search(r'<p[^>]*>(.*?)</p>', a, re.S)
    stars = re.search(r'aria-label="([\d,]+) stars today"', a)
    repos.append({
        "name": name,
        "desc": re.sub(r'<[^>]+>', '', desc.group(1)).strip() if desc else "",
        "today_stars": int(stars.group(1).replace(",", "")) if stars else 0,
    })

repos.sort(key=lambda r: r["today_stars"], reverse=True)
print(json.dumps(repos[:10], indent=2))
Enter fullscreen mode Exit fullscreen mode

That's the entire scraper. 25 lines. No BeautifulSoup, no Selenium, no Playwright — the standard library is enough because GitHub's Trending markup is remarkably stable.

The automation: GitHub Actions as a free cron server

This is the part I love. Instead of renting a VPS or running a Raspberry Pi, I use GitHub Actions as a free, always-on cron scheduler:

name: Daily Trending Deploy
on:
  schedule:
    - cron: '0 */6 * * *'   # every 6 hours
  workflow_dispatch:        # manual trigger

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python scripts/fetch_trending.py
      - uses: peaceiris/actions-gh-pages@v4
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./site
Enter fullscreen mode Exit fullscreen mode

A few things I learned the hard way:

  1. Use a dedicated deploy action. peaceiris/actions-gh-pages handles the branch switch and token permissions so you don't have to write git-push plumbing inside your workflow.
  2. Output a JSON file alongside the HTML. I generate both index.html and trending.json — the JSON endpoint means other tools (dashboards, bots, my own scripts) can consume the same data without re-scraping.
  3. Schedule drift is real. The */6 cron is reliable but not instant; if freshness matters, add a workflow_dispatch fallback and a manual "refresh now" button in the UI.
  4. Keep the scraper resilient. Wrap the fetch in try/except and keep the last good snapshot if GitHub ever changes its markup. An aggregator that serves yesterday's data is still useful; a blank page is not.

Why "no API" is a feature

Most devs reach for an API first. But when the source is a public HTML page, parsing it directly has real advantages:

  • No token to rotate, no rate limit to negotiate — you are a normal browser request.
  • No vendor lock-in — if GitHub changes something, you fix one regex, not a dependency version.
  • Runs anywhere — the script needs only Python 3, so it works on a laptop, a $5 VPS, or a GitHub Actions runner.

Where this pattern scales

The "scrape → transform → publish to Pages on a schedule" pipeline is a template, not a one-off. I've reused it for:

  • 📰 AI tool daily digests (RSS feeds)
  • 📈 Stock/crypto price trend monitors — same Actions cron, same Pages hosting, different source
  • 🎯 Hacker News top-story aggregation
  • 🛠️ DevOps toolchain watchlists

If you're a developer who wants a personal "tech radar" page that updates itself, this is a 1-hour weekend project.

Try it yourself

The full, production-ready version (with a responsive mobile-friendly site, structured JSON output, and the complete Actions workflow) is open source:

👉 https://github.com/Felixwang007/github-daily-trending

Live demo: https://Felixwang007.github.io/github-daily-trending

It's MIT licensed — fork it, rip out my content, and point it at whatever you want to track. If you build something cool on top of it, I'd genuinely love to hear about it in the comments or via a PR.

Also check out my other open-source project if you trade A-shares: A-Share Stock Analysis Expert — a three-pillar stock screening system that combines technical, fundamental, and sentiment analysis into one tool.

Top comments (0)