Running a browser games portal at scale sounds fun until you think about the maintenance. We host tens of thousands of HTML5 games, each one an embedded game from a different developer, a different codebase, a different era of web technology. Games break. Embeds go down. Third-party hosts change their URLs. And every broken game is a player who bounces and never comes back.
Manual QA at this scale is impossible. Nobody is clicking through 56,000 games to check they still load. So we built an automated QA pipeline that catches broken embeds before players do. Here is how it works.
The problem with embeds
A browser game embed is usually an iframe pointing at a game file hosted somewhere. That "somewhere" is the problem. The game might live on our own CDN or on a developer's server, and either way things change: SSL certificates expire, hosts reorganize their file structures, games get taken down, hotlinking gets blocked. From the player's perspective, all of these look identical: a blank rectangle where a game should be.
We needed a system that could detect all of these failure modes automatically, distinguish between "game is broken" and "game is slow," and do it across tens of thousands of pages without melting our servers.
The crawler
The core of the pipeline is a headless browser crawler built on Playwright. Every night, it works through the game catalog in batches:
- Load the game page and wait for the embed iframe to appear in the DOM.
- Check the iframe source returns HTTP 200. A 404 or 403 here is an instant flag.
- Wait for first paint inside the iframe. This is the key check. A game that loads its HTML shell but never renders a canvas is broken for all practical purposes, even if every HTTP request succeeded.
- Listen for console errors. A flood of uncaught exceptions during load usually means a broken dependency.
- Screenshot and diff. We capture a screenshot of the game area and compare it against the last known-good screenshot. A sudden change to a blank or error page is a strong broken signal.
The screenshot diff was the breakthrough. HTTP checks alone missed about 30 percent of real failures, games that loaded fine technically but rendered nothing. Pixel comparison catches those.
Handling false positives
The naive version of this system flagged hundreds of games a night, and almost all of them were fine. Games are slow. Ad networks time out. Third-party analytics scripts fail in ways that do not affect gameplay. We had to teach the system the difference.
Our rules now: a game is only flagged if it fails the same check three nights in a row. Console errors from known ad and analytics domains are ignored. Load timeouts are retried with a longer window before flagging. This brought false positives down to a handful per week, each one worth a human look.
The alerting loop
Flagged games land in a review queue with the screenshots, the error logs, and the last known-good date. A human reviews the queue each morning, and the common outcomes are: fix the embed URL, replace the game file from our archive, or delist the game if the developer's host is permanently gone.
The whole pipeline runs on a single modest server. Playwright with a concurrency of 8, batched over the night, covers the full catalog in about 6 hours. Total cost is trivial compared to even one part-time QA person, and it never gets tired or bored.
What I would do differently
If I were building this again, I would add real user monitoring alongside the synthetic checks. Our crawler loads games from a datacenter in one region; players hit us from everywhere, and regional CDN issues are invisible to us until someone complains. RUM would close that gap.
I would also version the game files we host ourselves. Right now a developer update can silently change a game's behavior, and our diff catches it, but we cannot roll back. Immutable versions with a rollback button is on the roadmap.
The takeaway
At catalog scale, quality is a pipeline problem, not a people problem. Automated checks with smart false-positive handling keep tens of thousands of embeds healthy with minutes of human attention per day. If you run any kind of embedded-content site, build the crawler before you need it.
We run all of this in production at gameziko.com, where every game in the library gets checked nightly. If you have built something similar, I would love to hear how you handle the false-positive problem.
Top comments (0)