Every Thursday, Epic gives away a game. Steam runs weekend freebies. Amazon Prime drops a batch that quietly expires two weeks later. Three tabs, three UIs, three different ways of saying "this offer ends soon."
I kept missing claims, so I built OpenClaim — an open-source tracker that aggregates all three, refreshes itself every four hours, and costs exactly nothing to run. No server, no database, no runtime API calls.
The architecture is boring in a good way, and I'll cover it below. But the part worth your time is what happened after it worked: three bugs that produced no errors, no failed builds, and no stack traces. The site kept serving pages. The data was just quietly wrong.
🔗 Live: seqat.github.io/OpenClaim
💻 Source: github.com/Seqat/OpenClaim (MIT)
The architecture, in one paragraph
There is no backend. A GitHub Actions cron job runs a Python pipeline every four hours, which writes a games.json file and commits it to the repo. GitHub Pages serves that file alongside a vanilla ES6 frontend. The browser fetches a same-origin static file — no CORS, no rate limits, no API keys, no cold starts.
GitHub Actions (cron, every 4h)
├─ GamerPower REST API ──┐
└─ Playwright → Amazon ───┤
▼
normalize + dedupe
▼
games.json ──▶ committed to repo
▼
GitHub Pages ──▶ browser
My first prototype called the GamerPower API directly from the browser. It worked on localhost and died immediately in production: no Access-Control-Allow-Origin header, and every visitor burning a request against a shared rate limit. Amazon was worse — there's no public API at all.
The fix is a pattern I'd now reach for by default on static hosting: push everything dynamic into the build step, ship the artifact. One scrape serves every visitor. As a bonus, the data ends up version-controlled — git log games.json tells me what was free last month.
Steam and Epic come from GamerPower's REST API, which already aggregates both. I didn't reinvent that. The part I actually had to build is Amazon.
The Amazon scraper, and one trick worth stealing
Amazon Luna / Prime Gaming renders its claims page client-side, so requests returns an empty shell. Playwright handles that. The interesting problem wasn't rendering, it was time: the listing page gives you titles and links, but expiration dates only exist on each game's detail page. Sixteen games meant sixteen extra page loads inside a CI runner.
Three things brought that down to a few seconds. First, five concurrent tabs behind a semaphore. Second, wait_until="domcontentloaded" instead of the default load. Third, the one I hadn't seen elsewhere — blocking assets, but fulfilling CSS instead of aborting it:
await page.route(
"**/*.{png,jpg,jpeg,webp,svg,css,woff,woff2,gif}",
lambda route: route.fulfill(status=200, body="", content_type="text/css")
if ".css" in route.request.url.lower()
else route.abort()
)
Images, fonts and media get aborted outright. CSS gets served an empty 200 response. The distinction matters: aborting a stylesheet request makes some SPA bundles throw during module loading, and the page never hydrates. An empty stylesheet satisfies the loader while transferring nothing.
Because the page hydrates asynchronously, waiting a fixed duration is either wasteful or flaky, so the scraper polls instead — reading body text every 250ms for up to 3.5 seconds and stopping as soon as a parseable date appears.
For selectors I leaned on attribute substrings (div[class*="Card"], [class*="title"]) rather than exact class names, since the build hashes them. It's more resilient than it looks, and less resilient than I'd like — more on that in a moment.
Bug 1: A ternary with two identical branches
Here's the line that shipped:
is_permanent = True if giveaway_type in ["game", "full game", ""] else True
Read it twice. Both branches return True. Every Steam and Epic entry — 31 of them — was flagged as "permanently free," rendering a ♾️ Permanent badge on the card even when the API had handed me a concrete expiration date sitting right there in the same object.
Nothing failed. No linter complained; it's syntactically fine and the variable is used. The tests didn't catch it because there were no tests. And it was invisible from the outside: a permanent badge on a game that expires next Tuesday looks exactly like a permanent badge on a game that doesn't.
The fix is the whole point of how boring it is:
end_date = parse_iso_date(item.get("end_date"))
is_permanent = end_date is None
The real lesson isn't "read your ternaries." It's that this field was derived state stored as a flag. The moment is_permanent became something you could set independently of end_date, it became something that could disagree with end_date — and nothing in the system was checking that they agreed. The one-line check I now run against generated output would have caught it on day one:
bad = [g for g in games if bool(g["is_permanent"]) != (g.get("end_date") is None)]
Bug 2: A broken scraper publishes an empty site
This one is baked into the architecture I just recommended to you, so it's worth dwelling on.
Both scrapers wrapped their work in a try/except that logged the error and returned an empty list. Reasonable in isolation — one source failing shouldn't take down the other. But main() then did this:
final_games = normalize_and_deduplicate(all_games)
write_json(final_games) # unconditionally
If Amazon changed a class name, or the runner hit a network hiccup, or GamerPower had a bad minute, the pipeline would cheerfully write an empty games.json, commit it, and push. The job goes green. The site shows zero games. Nothing alerts you, because from CI's perspective everything succeeded — the code ran to completion and produced a file.
This is the failure mode nobody mentions when they tell you to move dynamism into the build step. A server that crashes is loud. A build that succeeds while producing garbage is silent, and it overwrites the last known-good artifact on its way out.
The guard is crude and works:
if not final_games:
logger.error("No free games fetched. Aborting write to prevent saving empty file.")
sys.exit(1)
if previous_games and len(final_games) < len(previous_games) * 0.5:
logger.error(
f"Sudden drop in free games count (fetched {len(final_games)}, "
f"previously {len(previous_games)}). Scrapers might be broken. Aborting write."
)
sys.exit(1)
Two properties I'd argue are non-negotiable for any commit-the-artifact pipeline:
-
Failing loudly beats writing quietly.
sys.exit(1)turns an invisible data loss into a red X in the Actions tab. - The previous artifact is your baseline. You're committing the output to git, so last run's data is right there. Comparing against it costs nothing and catches partial breakage that an emptiness check misses.
The 50% threshold is arbitrary and I'll probably regret it the first time a legitimately quiet week trips it. That's a better failure than the alternative.
Bug 3: A regex that was too helpful
Amazon writes expiration dates in whatever format it feels like — Available through Sep 23, a localized Turkish string, sometimes a relative in 12 days. So the parser tries patterns in sequence, and the last one was a catch-all:
re.search(r"(\d+)\s*(?:gün|days?)", text)
Applied to page.inner_text("body"). The entire page.
Think about what else lives on an Amazon game page. Review timestamps. Description copy. And, memorably, game titles — 7 Days to Die would have handed that regex a very confident 7, which becomes "expires in 7 days," which becomes a countdown timer on a card, which a user believes.
I deleted the fallback. None is a correct answer; a fabricated date is not. The card renders without a countdown and nobody is misled.
There was a subtler issue in the same code path. Relative dates were computed as now + timedelta(days=n) at scrape time — so a game "expiring in 12 days" got a slightly later timestamp on every four-hour run, drifting forward forever. Pinning to .replace(hour=23, minute=59, second=59) makes the value stable across runs.
The pattern behind all three: in scraping, flexibility is indistinguishable from fabrication. Every fallback you add to "handle more cases" is also a fallback that will confidently produce a wrong answer instead of admitting it doesn't know.
What the deduplication taught me about keys
Minor compared to the above, but it bit in a way I found instructive.
The original dedupe key was a normalized title. Which quietly meant that when a game was free on Steam and Epic in the same week — which happens more than you'd think — one of them silently vanished from the site. The dedupe was working perfectly. The key was just wrong: two entries with the same title on different stores aren't duplicates, they're two different deals a user might want.
norm_title = re.sub(r"[^\w\s]", "", title.casefold())
norm_title = re.sub(r"\s+", " ", norm_title).strip()
norm_key = (platform, norm_title)
Two details in there earned their place. re.sub(r"\s+", " ", ...) because without it "Game Name" and "Game Name" hash apart. And casefold() rather than lower() — the site is bilingual, and Turkish has a dotless ı, so lower() gives you a different string than you expect for titles containing I.
Where it stands
The pipeline runs every four hours, plus two extra attempts on Thursday afternoon to catch Epic's weekly rotation — GitHub's cron is delayed under load often enough that a single scheduled run isn't reliable if you care about a specific moment.
There's a small pytest suite now. Nine tests, which is not many, but they cover the three functions that actually produce data: date parsing, title cleaning, deduplication. One of them exists purely to fail if anyone reintroduces that catch-all regex.
Known rough edges, since you'll find them anyway:
- GamerPower returns DLC, playtest access and in-game bundles alongside actual games. They're all technically free offers; they're also noise if you came looking for games. A type filter is the next patch.
- The Amazon scraper depends on class-name substrings. It will break. The guard above means it breaks loudly instead of publishing an empty page, which is the best I've got short of a proper contract test.
- Tests don't run in CI on pull requests yet.
-
generated_atchanges on every run, so the pipeline commits even when the game list is identical. Six commits a day of pure noise.
If you've run a scraper aggregator for any length of time, I'd genuinely like to know what you did about breakage detection. Health-check endpoints, schema contracts, canary records, or just staring at the Actions tab like I do?
Adding a platform is four steps and documented in CONTRIBUTING.md — drop a file in backend/scrapers/, return a list of dicts matching the schema, register it in main.py. GOG and itch.io are the obvious next ones.
Top comments (0)