What I learned using git rev-list to diff a data file across a seven-day window

The conclusion first: if a data file you work with is committed to git and refreshed on a schedule, git rev-list gives you its state from any past window at zero infrastructure cost. You don't need a checkpoint file or a second cron job to save snapshots. The pattern has real edge cases — shallow clones, squash merges, force pushes — but they're predictable and handleable.

Why I needed a weekly diff in the first place

The HuggingFace models dataset powering this project's AI tools directory refreshes daily. Each run pulls the top models by downloads, updates apps/ai-tools/src/data/models.json — a JSON array of roughly 8,000 model objects — and commits the result to main. The file is committed on every successful ETL run.

Once a week, I publish a "new models this week" article. The question is: which model slugs are in today's models.json that weren't there seven days ago?

Counting by API createdAt doesn't work. Four filters I apply when pulling HuggingFace models into an AI tools directory explains why: the filter criteria (minimum downloads, pipeline tag filters) change over time. A model created six months ago can first appear in my filtered dataset today. What I need is appearance in the dataset, not creation date.

The obvious approach is to store a checkpoint. At the end of each week, copy models.json to models-prev.json, commit it, compare next time. This works but adds a file to maintain, requires a scheduled job to run at exactly the right time, and introduces a silent failure mode: if the snapshot cron doesn't run, the next diff is stale with no indication.

What git rev-list gives you

git rev-list walks the commit graph and returns commit SHAs in reverse chronological order. The --before= flag filters to commits older than a given timestamp. A path argument restricts to commits that touched a specific file.

BASE=$(git rev-list -1 --before='7 days ago' origin/main -- apps/ai-tools/src/data/models.json)

This returns the SHA of the most recent commit to models.json that was made more than seven days ago. The -1 flag limits to one result. The path after -- means only commits that actually modified this file are considered — commits to other files don't advance or retract the baseline.

Then:

git show "$BASE:apps/ai-tools/src/data/models.json" > /tmp/models-prev.json

git show <sha>:<path> outputs the exact file content at that commit without touching the working tree or moving HEAD. It's a clean read. No checkout, no stash, no branch switching.

What you get is the state of your data file seven days ago, reconstructed from git's object store, using history that already exists. No second file needed. No additional cron.

The full production pattern

Here's the complete version I use in the article-generation routine:

BASE=$(git rev-list -1 --before='7 days ago' origin/main -- apps/ai-tools/src/data/models.json)
git show "$BASE:apps/ai-tools/src/data/models.json" > /tmp/models-prev.json \
  || cp apps/ai-tools/src/data/models.json /tmp/models-prev.json

python3 -c "
import json
old = {m['slug'] for m in json.load(open('/tmp/models-prev.json'))}
new = [m for m in json.load(open('apps/ai-tools/src/data/models.json')) if m['slug'] not in old]
new.sort(key=lambda m: (m.get('downloads') or 0), reverse=True)
print(json.dumps(new[:8], indent=1))
"

Three parts worth separating:

Baseline SHA. git rev-list -1 limits to one result. --before='7 days ago' anchors to calendar time, not commit count. The path argument -- apps/ai-tools/src/data/models.json means only commits that actually changed this file count — intervening commits to other files don't move the baseline.

Baseline file extraction. The || cp ... fallback handles two cases where $BASE is empty: the repository history is shallow (no commit predates seven days), or the file didn't exist yet. When the fallback fires, the diff produces zero new entries, which is the correct behavior — there's nothing to report.

Python diff. Set subtraction on slugs. Sort by downloads descending to surface the most-used new models first, since that's what a "new this week" reader cares about. The m.get('downloads') or 0 guard handles models where the field is null. Three slug-collision strategies I evaluated for a HuggingFace model directory ETL — slug is the stable identity here, not model name, which can change without a new repo being created.

When the pattern breaks

Shallow clones. GitHub Actions uses actions/checkout@v4 with fetch-depth: 1 by default, pulling only the latest commit. git rev-list --before='7 days ago' finds nothing in a depth-1 clone. Fix: set fetch-depth: 0 in the checkout action, or use a fetch-depth of at least 30 for a month of daily commits.

- uses: actions/checkout@v4
  with:
    fetch-depth: 0

This is the edge case most likely to bite you silently. Three ETL failure patterns I now write into the output file, not just the logs — the shallow-clone failure is exactly the class of failure that produces zero output without any error message, making it look like "no new models this week" when the baseline simply wasn't found.

Squash-and-merge workflows. If your main branch uses squash merge, a week of ETL runs collapses to one commit per merged PR. The --before='7 days ago' filter might land on a commit from three weeks ago rather than one week ago. The result is a wider diff than intended — more "new" models than actually arrived this week. For data files committed directly to main by a cron (as mine are), squash merge doesn't apply. But if your ETL lands data through PRs, this is a real distortion.

File path changes. If the file moved or was renamed within the past seven days, git rev-list -- old/path returns nothing. git log --follow handles renames; rev-list alone doesn't. Track renames explicitly if this could happen to your file.

Force pushes to main. If someone rewrote history on the branch you're targeting, commit timestamps shift. --before='7 days ago' may select a rewritten commit whose original date is no longer meaningful. I prevent force pushes via branch protection, but if your setup doesn't, the pattern is silently unreliable after any rewrite.

Verifying the baseline before trusting the diff

One thing the pattern above doesn't show: I check that the baseline commit is recent before treating the diff output as "this week." If $BASE points to a commit from three weeks ago because nothing touched the file in eight days, a naive article claims those models are new this week.

if [ -n "$BASE" ]; then
  BASE_DATE=$(git show -s --format='%ci' "$BASE")
  DAYS_OLD=$(( ( $(date +%s) - $(date -d "$BASE_DATE" +%s) ) / 86400 ))
  if [ "$DAYS_OLD" -gt 9 ]; then
    echo "WARN: baseline is ${DAYS_OLD} days old — data file may not have changed this week"
  fi
fi

Three approaches I use to catch silent failures in a cron-heavy GitHub Actions pipeline — this is the same failure category: a job completes successfully but produces wrong output because a dependency silently degraded. Writing the baseline date to the output file makes the issue visible in the article-generation log.

What I considered and didn't use

A dedicated "last week snapshot" file. Simplest to understand, adds a file to maintain and a cron to run reliably. Four GitHub Actions cron scheduling patterns in a monorepo — adding another cron to an already-dense schedule was something I wanted to avoid.

Comparing createdAt from the HuggingFace API. The API returns createdAt as the date the model repo was first created, not when it entered my filtered dataset. A model created six months ago can first appear in my downloads-filtered set today after a spike in adoption. Three public HTTP APIs I read daily without registering for a key — using the API's timestamp to mean "new to my dataset" requires the API's semantics to align with mine, which they don't here.

Storing first_seen_at in libSQL and querying it. Four libSQL queries I use to catch ETL gaps in my AI model directory — I do store first_seen_at in the database and could query it. But the article-generation routine runs without database credentials for simplicity, and adding a database dependency to a script that previously needed only a git checkout was a tradeoff I wasn't willing to make.

The git approach won because it requires no infrastructure changes and works anywhere the repo is cloned with sufficient history.

What would make me stop using it

If the ETL moves to squash-merge PRs to land data to main, --before='7 days ago' becomes unreliable for this use case — the baseline window could be months wide.

If models.json grows large enough that git show is slow on every article run, I'd evaluate the first_seen_at database query as the replacement. Currently the file is around 3MB and git show completes in under a second.

If HuggingFace adds a stable "models added this week" endpoint with documented semantics and a reliable timestamp, I'd use it instead. How I detected deleted YouTube videos using JSONL history diffing — git-as-diff-source is a workaround for missing API functionality, and a good-enough API endpoint is always preferable. The git rev-list documentation covers the full option set including --before, --after, and --count if you want to extend this pattern.

Frequently asked questions

Does this work if models.json is in Git LFS?

Only if you've fetched LFS content. actions/checkout doesn't fetch LFS by default — add lfs: true to the checkout step. Without LFS content, git show returns a pointer file, not the JSON, which silently corrupts the diff. My models.json is not in LFS, but this would be a hard-to-debug failure if it were.

Can I target a specific date instead of a relative one?

Yes. --before='2026-08-13T00:00:00Z' accepts ISO 8601 timestamps. I use relative '7 days ago' because the article runs weekly and "this week" should always mean the past seven calendar days from run time. A fixed date would require updating the script each week.

What if the file didn't change in the past seven days?

git rev-list -1 --before='7 days ago' returns the most recent commit before the cutoff that touched the file. If the file wasn't modified for two weeks, the baseline is from two weeks ago, and the diff shows more "new" entries than intended. The baseline-age check above catches this and logs a warning.

What if I need this on a branch, not main?

Replace origin/main with any ref: origin/my-branch or HEAD for the current branch. I target origin/main explicitly to decouple the baseline from whatever branch the script runs on — this matters when the article-generation routine runs on a feature branch during testing.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.