MORINAGA

Posted on Jul 7 Edited on Aug 27

How I added upstream activity signals to an OSS alternatives directory

#showdev #opensource #webdev #typescript

Three months into running ossfind.com — my open-source alternatives directory — I got a soft rejection from AdSense citing thin programmatic content. The full breakdown of what triggered it and what I changed is its own post, but one piece of the response was a new ETL script I shipped last weekend: etl-evidence.mjs, which pulls GitHub activity signals for every curated alternative on the site.

The idea: each page now shows the latest stable release, how many releases shipped in the past 12 months, open issue and PR count, and whether the repo is archived. Factual, not interpretive. This post covers the implementation and the non-obvious decisions that made it more than a loop over fetch() calls.

Why activity signals, specifically

Programmatic directory pages are thin for an obvious reason: they're assembled from API metadata, not from someone who actually used the software. The intro for "alternatives to Datadog" isn't wrong — but it reads like a database dump. The missing piece is proof that these projects are real and alive.

What I wanted was something that answers the reader's actual question before they click through: "Is this project maintained?" That answer exists in the GitHub API and it's objective. A project with a stable release two weeks ago, 600 open issues, and 180 PRs is clearly active. A project with a release from 2019 and archived status tells a different story. Neither of those judgments requires AI inference — the numbers carry the signal.

The alternative I rejected was asking Claude Haiku to generate a one-sentence "Our take" on project health from those same signals. I deleted that version of the script before committing. The reason: auto-generated opinion text is the exact pattern that got the site flagged in the first place. The E-E-A-T transparency pages I added earlier already explain the methodology; each page already has hand-written editorial takes. Adding more AI-generated summary text on top of facts would loop back to the original problem.

Which repos the ETL fetches

The ETL runs for curated pages only — the subset that passed the quality bar high enough to be visible in Google. The isCurated() check mirrors the condition in src/lib/curation.ts:

const isCurated = (s) =>
  s.intro && s.intro.length >= 80 &&
  !(s.model_used && GENERIC.has(s.model_used)) &&
  (s.alternatives ?? []).length >= 3 &&
  (s.alternatives ?? []).reduce((m, a) => Math.max(m, a.stars ?? 0), 0) >= 800;

Pages that don't meet this threshold are behind the noindex gate anyway. Running the evidence ETL across every repo referenced in saas.json — 80 SaaS rows, 206 alternative associations, 150 distinct repos — would spend API calls on pages that aren't indexed. The comment in the script reads ⚠️ KEEP IN SYNC: src/lib/curation.ts CURATION — the bluntest way I know to flag a coupling that can drift.

The latest run wrote evidence for 107 repos. At 2 API calls per repo (one for repo metadata, one for releases), that's 214 requests against a 5,000/hour limit — comfortable margin.

Handling failure modes differently

The failure handling was the most substantive design work. There are three failure modes, and each one calls for a different response.

Rate limit or auth errors (HTTP 403 or 429) affect the whole session. If you hit one mid-batch and keep going, you write null or stale data for every remaining repo. On the next build, those records become the "current" evidence. The result: a CI run with a bad or expired GITHUB_TOKEN could silently blank out evidence cards across the site.

The fix is a typed exception class:

class RateLimitError extends Error {}

async function gh(path) {
  const r = await fetch(`${API}${path}`, { headers });
  if (r.status === 404) return null;
  if (r.status === 403 || r.status === 429) throw new RateLimitError(`${r.status} ${path}`);
  if (!r.ok) { console.error(`  GH ${r.status} ${path}`); return null; }
  return r.json();
}

The main loop wraps in a try/catch that catches only RateLimitError, logs it, and calls process.exit(1) without writing the output file. The existing evidence.json stays on disk untouched. The build picks it up and continues with last-run data.

Individual 404s mean a specific repo was deleted or renamed — normal for open-source over multi-year timescales. Here, aborting the batch would be wrong. The right response is to preserve whatever data the previous run fetched for that repo and move on. The ETL reads existing data into existing at the start of the run:

let existing = {};
try { existing = JSON.parse(readFileSync(OUT, "utf8")).repos ?? {}; } catch { /* first run */ }

When a fetch returns null (the 404 path), it checks existing[repo] and copies it to the output if present. This keeps a deleted repo's last-fetched data visible until someone removes it from the source JSON manually — not ideal long-term, but better than a sudden blank field.

Other transient HTTP errors (500, 502, etc.) log to stderr and return null. The next run retries. No special handling needed.

Release data: three edge cases

The releases endpoint has more nuance than the repo metadata endpoint.

Pre-releases and drafts. The endpoint returns everything — alphas, betas, release candidates, drafts. Showing a pre-release as "latest release" would mislead visitors who assume that label means stable. I filter before picking the top entry:

const stable = rels
  .filter((r) => !r.draft && !r.prerelease && r.published_at)
  .map((r) => ({ tag: r.tag_name, at: r.published_at }))
  .sort((a, b) => b.at.localeCompare(a.at));

For projects that only do pre-releases (some large frameworks in active development), latest_release_tag ends up null and the component simply omits the "Latest release" line — the other signals still render. Printing an explicit "no stable releases tagged" line would be more useful context, and I haven't added one yet.

The 100-per-page cap. GitHub's releases endpoint returns at most 100 items per page. The ETL doesn't paginate. For a project shipping 3–4 releases per month that's harmless — 100 items reaches back two years or more, so the last 12 months are fully covered. It bites for projects whose most recent 100 releases all fall inside the last 12 months. The flag I set for that is coarser than the problem:

releases_last_year_capped: rels.length >= RELEASE_PAGE && releasesLastYear >= RELEASE_PAGE,

releasesLastYear is counted after dropping drafts, pre-releases, and anything published more than a year ago. So the flag only goes true when all 100 items the API handed back are stable releases from the last year — 6 of the 107 repos in the current run. A project that ships 100+ times a year with release candidates or drafts interleaved gets a page-capped, under-reported count and no 100+ label to admit it. Real pagination is the fix and I haven't written it yet.

When releases_last_year_capped is true, the page renders "100+ releases" rather than "100." The distinction matters for projects like Neovim or Nix — the fact that they ship constantly is the signal, not the exact count.

Archived repos. An early version of the component said "This project is no longer maintained" when the archived flag was true. That's a judgment. Some repos are archived because they're finished and stable, not because they're abandoned. I changed it to "Repository archived on GitHub (read-only)" — factual, not editorial, consistent with the E-E-A-T methodology that explains what these signals mean.

Wiring into the refresh pipeline

There's no dedicated workflow for this. The script is chained onto the end of the app's own refresh script, after the main ETL and export:

"evidence": "node scripts/etl-evidence.mjs",
"refresh": "pnpm run seed && pnpm run etl && pnpm run generate && pnpm run export && pnpm run evidence",

refresh-content.yml then runs pnpm run refresh per app through its matrix, so the evidence step inherits the workflow's GITHUB_TOKEN and its schedule without any new YAML. The secrets.GITHUB_TOKEN installation token is the correct choice here — it's scoped to the repository, rotates automatically, and gives the 5,000 request/hour limit for authenticated calls. I don't need a PAT. The four GitHub Actions patterns I use across the monorepo cover why I prefer GITHUB_TOKEN over PATs for refresh workflows.

The isCurated() filter was in the script from the first commit, and it's the thing that keeps the cost sane: without it the loop would cover all 150 distinct repos in saas.json instead of the 107 on curated pages, at 2 calls each — roughly 86 wasted calls per daily run producing data nobody would see.

What I'd change

Store the raw response alongside derived fields. Right now the ETL writes only open_items, latest_release_tag, latest_release_at, releases_last_year, releases_last_year_capped, and archived — star counts come from the main SaaS dataset, not from here. When I wanted to add "has_discussions" later, I had to go back to the API. If I'd kept raw TEXT in the output JSON, I could have read it without a new network call.

A deprecated_at field for gone repos. Currently a 404 silently preserves old data forever. A timestamp on when the fetch last failed would give the content team something actionable: "this entry hasn't verified in 90 days, check if the repo moved."

Weekly cadence for releases, daily for issue counts. Open issue counts shift daily; release dates don't change that fast. Two separate cron jobs with different frequencies would halve the API calls with no visible difference to readers.

FAQ

What if a project has no GitHub releases?

latest_release_tag and latest_release_at return null, and the component drops the "Latest release" line entirely rather than printing a placeholder. Many active projects distribute via npm or PyPI without tagging GitHub releases, so the absence isn't itself a negative signal — but saying so explicitly would be better than saying nothing, and that's on my list.

Why not paginate releases to get an accurate count?

For this use case, precision past 100 doesn't matter. A project that ships 120 releases per year and one that ships 80 are both "very actively maintained." The "100+" display handles the clean version of the ambiguity, and as noted above it misses the messier one. Pagination would double or triple the API call count for the repos where it matters, which is why I've left it.

Can the ETL run locally without a token?

It exits immediately with an error if GITHUB_TOKEN isn't set. There's no useful fallback for a live fetch script. The last-written evidence.json stays intact, so a local build still works fine — it just uses whatever data the last CI run produced.

How does this fit with the three-tier content model?

The three-tier quality ladder applies to editorial text — seed, fallback-template, Claude Haiku. Evidence signals sit on top of that ladder rather than beside it: isCurated() treats fallback-template and seeded-from-json as generic, so the ETL never spends API calls on repos that only appear on those pages. Rendering is a separate question: the template shows a card whenever evidence[a.repo] exists, keyed by repo rather than by page, so a non-curated page that happens to list a repo already fetched for a curated page still renders its signals — 17 of the 48 non-curated pages do today. The fetch side is deliberate — the pages that most need a credibility boost are exactly the ones I don't want to dress up with extra chrome.

DEV Community