DEV Community

kairi
kairi

Posted on

Building a GitHub Trending Tracker Under D1's 100k Writes/Day Limit

I wanted a quick way to answer "what is taking off on GitHub right now?" Sorting by total stars mostly returns projects that have been popular for years. So I built OSS Today, a small site that ranks repositories by the stars they gained in a period you choose.

OSS Today home page

What it does

  • Rising: ranks repositories by stars gained in the last 24 hours, 7 days or 30 days.
  • New: repositories created in the selected period, ranked by total stars.
  • By language and by framework: 15 language pages and 24 framework pages (React, Django, and so on), plus a combined filter.
  • Good first issues: keeps only repositories that have at least one open issue labeled "good first issue". I use this to find somewhere to contribute.
  • Contribute panel on every card: links to the CONTRIBUTING guide, open issues, discussions and the fork page.
  • An RSS feed of the weekly Rising ranking, and a Japanese version under /ja.

There is no sign-up. The "Updated" label next to each ranking tells you how fresh the data is, and the info icon next to it explains how the numbers are computed.

How the Rising ranking works

GitHub's API gives you the current star count of a repository, but not its history. To know how many stars a repository gained in the last week, you have to record the count yourself.

So the site keeps a pool of up to 4,000 repositories in a D1 database and stores a star-count snapshot whenever a repository's count changes. The gain for a period is:

gain = current stars - stars at the start of the period
Enter fullscreen mode Exit fullscreen mode

For a repository created during the period, all of its stars count as gained.

A Cloudflare cron job runs every 10 minutes:

  1. Refresh: it re-checks the repositories that are due, up to 100 per run, with GraphQL nodes() lookups.
  2. Discover: every other run, it searches GitHub for new candidates, both new repositories and established ones that were pushed to recently.
  3. Prune: when the pool is over its limit, the slowest-growing repositories are dropped.

Not every repository needs the same attention, so re-check frequency depends on how visible a repository is. Anything in the top 10 of any ranking is re-checked about every 2 hours. Other repositories gaining at least 10 stars a day are re-checked every 6 hours, ones gaining at least 10 a week every 12 hours, and the rest about once a day.

Bar chart of how many of the 4,000 tracked repositories fall into each re-check interval: 511 every 2 hours, 1,100 every 6 hours, 1,322 every 12 hours, 1,067 every 24 hours

Once an hour, one pass over the pool rebuilds every ranking (3 periods, with and without the good-first-issues filter, overall plus every language and framework) and stores each list as a single row. A page view reads one ranking row and the few dozen repositories it displays.

Staying inside the free tier

The whole thing runs on Cloudflare Workers (through the OpenNext adapter for Next.js) and D1, on the free plans. D1's free plan allows 5 million rows read and 100,000 rows written per day, and I hit both limits in the first week.

  • Reads. My first version scanned the whole pool on every ranking cache miss, and the edge cache is per location, so reads grew with visitors and crawlers instead of with the data. The account reached 81% of the daily read limit before 10:00 UTC. Precomputing the rankings once an hour fixed it: reading a page now costs on the order of 60 to 330 rows instead of thousands.
  • Writes. After that, writes became the tighter limit. I moved to one row per ranking instead of one row per list entry, dropped an index that cost an extra write per refresh, ran discovery less often, and re-checked stagnant repositories daily instead of every 12 hours.

Horizontal bar chart of D1 rows written on Nov 9 by query type: refreshing repo rows 14,828, star-count snapshots 11,804, deleting snapshots 5,810, adding discovered repos 2,538, pruning repos 2,196, and three smaller ones

On a typical day now, the site reads about 1.2 million rows (roughly 25% of the limit) and writes about 35,000 (roughly a third). Most of what I learned was from measuring production, not from reasoning about the code. For example, my estimate of how often repositories could be re-checked turned out to be wrong, because the refresh budget (100 repositories per run) was smaller than what my schedule asked for. Allocating that budget by visibility, as described above, is the result.

![Line chart of D1 usage as a share of the free daily limit from Nov 1 to Nov 9: rows written stay near 33 to 36 percent, rows read near 22 to 27 percent, with one day at 53 percent on Nov 3](ADD_CHART_URL_d1-usage](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/wg6xjvilrbaryxyaf6wy.png)

Limitations

  • Only tracked repositories can appear. The pool holds up to 4,000, so a repository that is growing fast but that I have not discovered yet will be missing for a while.
  • For a repository I started tracking after the period began, the gain is measured from my first recorded count, so it can be understated. The info dialog on the site says this too.
  • Star counts are not a quality signal, and stars can be bought. This is a discovery tool, not a recommendation.

Try it and tell me what is missing

The site is at osstoday.dev. I would especially like to hear:

  • which repositories you expected to see and did not,
  • which languages or frameworks to add,
  • whether the "gained in a period" ranking matches what you would call trending.

I plan to keep updating OSS Today, and your feedback will shape what comes next.

Thanks for reading.

Top comments (0)