Building a Self-Healing GitHub Trending API in One Day

Pool Guard — Sun, 19 Jul 2026 01:08:36 +0000

What happens when you refuse to accept "the source is down" as an answer.

The problem
GitHub's trending page is where millions of developers find new projects every day. But GitHub has never shipped an official API for it — if you want that data programmatically, you have to scrape the HTML.

Every existing scraper I found had the same design: one source, one point of failure. Small HTML change from GitHub → your service is 503 for a week. Rate limited from Cloudflare → same result.

I wanted a trending API that stays up when GitHub's page is down. So I built Hydra, and put it live at hydra9.dev.

The architecture (in one picture)
┌───────────────┐
Request ───▶ │ Router │
└───────┬───────┘
│ tries in order, breaker-guarded
┌───────────────┼────────────────┬─────────────────┐
▼ ▼ ▼ ▼
┌────────┐ ┌──────────┐ ┌──────────────┐ ┌──────────────┐
│Track A │ │ Track B │ │ Track C │ │ Track D │
│HTML │ │Community │ │Search API │ │gitstar- │
│scrape │ │mirror │ │(created:>7d) │ │ranking.com │
└────────┘ └──────────┘ └──────────────┘ └──────────────┘
Any track dead? Circuit breaker opens, next request routes past it.
All tracks dead? Serve most recent Postgres snapshot + fire Prometheus alert.
Four independent sources. A track that fails 3 times in a rolling window gets its breaker opened for 60 seconds — the router jumps to the next track without adding latency to the user's request.

Why 4 sources and not 2
Two sources feels like enough right up until they share a failure mode. The four I picked are architecturally independent:

Track A (github.com/trending HTML): freshest data, but the format changes without warning
Track B (ghapi.huchen.dev community mirror): a nice person maintains this, but it can go quiet for hours
Track C (GitHub Search API created:>7d sort:stars): the most stable, but it's reconstructed trending — not what appears on the actual page
Track D (gitstar-ranking.com): different data model entirely (lifetime stars), useful only as a last-resort cross-check
If A breaks because of a GitHub HTML change, B doesn't (it has its own parser). If B breaks because the maintainer's server goes down, C is a completely different infrastructure at GitHub. If C rate-limits you, D is a totally different site.

Historical data — the thing GitHub won't give you
The official trending page is a snapshot in time. If you refresh, yesterday's list is gone forever.

Hydra runs a Celery beat task every 15 minutes that stores a fresh snapshot to Postgres. That unlocks four analytics endpoints the official page can't offer:

Repos that showed up today but weren't there yesterday

curl https://hydra9.dev/api/v1/trending/new-stars

Repos that just fell off the list

curl https://hydra9.dev/api/v1/trending/dropped

Ranked by stars/hour instead of cumulative fame

curl https://hydra9.dev/api/v1/trending/velocity

Look at any point in time

curl https://hydra9.dev/api/v1/trending/history?at=2026-07-14T00:00:00Z
velocity in particular has been my favorite — it surfaces projects that are exploding right now, not projects that were famous 6 months ago and still coasting.

The stack (deliberately boring)
API: FastAPI (Python 3.11+)
Task queue: Celery + Redis (broker on db 1, results on db 2)
Storage: PostgreSQL 16 + Alembic
HTTP client: httpx (async everywhere)
HTML parser: selectolax (faster than BeautifulSoup)
Observability: Prometheus + Grafana + 7 alert rules
Deploy: Docker Compose (single file)
The whole thing is 52 Python files and 47 tests that run in 6 seconds.

One lesson: @lru_cache and Pydantic don't mix
The single bug that took me from "green in dev" to "500 in prod" was this innocent-looking function:

@lru_cache
def track_singletons(settings: Settings) -> dict[TrackId, Track]:
return {
TrackId.A: TrackA(settings),
TrackId.B: TrackB(settings),
TrackId.C: TrackC(settings),
TrackId.D: TrackD(settings),
}
Pydantic's BaseSettings is mutable and doesn't implement __hash_. lru_cache needs hashable arguments. Every real request threw:

TypeError: unhashable type: 'Settings'
Fix: since get_settings() is itself an @lru_cache'd singleton, the parameter was redundant. Call it inside:

@lru_cache
def _track_singletons() -> dict[TrackId, Track]:
settings = get_settings()
return {...}
Silly bug. The tests didn't catch it because they patched Settings with a mock that was hashable. Lesson: your test doubles should mirror the real thing's flaws, not paper over them.

Try it
Live: https://hydra9.dev
API docs: https://hydra9.dev/docs
Source (MIT): https://github.com/loly-baby/Hydra
Self-host in 4 commands:
git clone https://github.com/loly-baby/Hydra
cd Hydra && cp .env.example .env
docker compose -f infra/docker-compose.prod.yml up -d --build
curl http://localhost:8000/api/v1/trending
If it's useful, a ⭐ on the repo goes a long way. If you have a use case where you'd want more analytics (release velocity? topic clustering? language-specific momentum?), open an issue.

Hydra runs on a single 6GB VPS shared with another project. If the public instance gets hammered, self-hosting is a Docker Compose away.

DEV Community: Pool Guard

Building a Self-Healing GitHub Trending API in One Day

Repos that showed up today but weren't there yesterday

Repos that just fell off the list

Ranked by stars/hour instead of cumulative fame

Look at any point in time