The pipeline already knows how healthy the project is. It just never tells anyone.
Nobody Reads CI Logs
Every push to main on my side project produces a pile of reports. Vitest JSON for unit and component tests. A Playwright report across three browser projects. Coverage. Type coverage. Lighthouse runs. Bundle size. Eval results. Then the run goes green, the artifacts expire after 90 days and nobody looks at any of it again.
As a QA Engineer, that bothers me more than it should.
So the last job in the pipeline now turns those reports into a page. One HTML file, built from the run's own artifacts, published to GitHub Pages next to the app it describes.
It's live here: https://newell-paul.github.io/sprint-scout/dashboard/
The App
The app is SprintScout, a browser-only planner that times cycling segment attempts to the wind.
https://newell-paul.github.io/sprint-scout/
It has some strict rules: no server, no accounts, no API keys, no request the rider didn't ask for.
The dashboard had to live by the same rules.
-
No scripts. Every chart is inline SVG with
role="img"and a label. The values stay in the page as text. - No external requests. No chart library, no CDN. Even the fonts are files sitting next to the page.
- No database. The history is a JSON file carried from one run to the next as an artifact.
- A missing input never fails the build. No Lighthouse report? That card says n/a. A broken report shouldn't hide a good release, and a missing number shouldn't pretend to be zero.
The whole thing is one TypeScript file, packages/testing/src/dashboard.ts
What's On It
Eight sections, each a grid of cards with a tone and a trend.
- Pipeline. Checks duration, first-pass success and the last 30 runs as bars. At time of writing: 4.5 minutes, 100 % first-pass (24 passed first time, 6 cancelled because a newer push superseded them).
- Tests. Counts per level: 701 unit in 5.5 s, 96 component in 9.9 s, 139 end-to-end in 106.8 s across three browsers. Plus a flake-free streak.
- Intent board. More on this below.
- Quality gates. Line coverage 92.7 %, branch 90.8 %, mutation score 81.0 % from the last Stryker run, type coverage 99.62 % against a 99.5 % gate.
- Performance. Lighthouse 96, 98, 100 and 91, LCP 628 ms, TBT 49 ms, 450 kB of client JavaScript against a 550 kB ceiling.
- AI practice. Evals, intents and review findings. More below.
- Delivery. PR cycle time and merges per week, read from the GitHub API.
- Slowest tests. The five slowest end-to-end tests. All five are WebKit accessibility runs, which is its own little story.
And badges. The README badges read small JSON files the dashboard publishes for shields.io, so they come for free.
Bands, Not Thresholds
A threshold tells you coverage dropped below 80 %. It doesn't tell you the end-to-end suite has been creeping up by four seconds a day for a fortnight.
So the dashboard has bands. Eleven of them, in a YAML file:
- id: e2e-duration
label: End-to-end duration
path: e2e.durationMs
worse: higher
unit: ms
minSigma: 5000
Each run on main is compared with the previous 20 days. The distance from their mean, in standard deviations and in the worse direction, sets the level:
- 1σ is logged in the job summary.
- 2σ annotates the dashboard.
- 3σ fails the dashboard job, so the run goes red and Pages doesn't publish it.
minSigma stops a flat baseline from screaming about noise. If the suite has taken 106 seconds every day for a week, the standard deviation is nearly zero and a one-second wobble would be a 10σ event. With minSigma: 5000 it has to move five seconds before it counts.
since handles an intended step change. Delete a pile of redundant end-to-end tests on purpose and the count drops. Set since to that date in the same commit and the old days leave the baseline.
A band stays inert until it has five days of history. Statistics on two data points is astrology.
The fun part is that a 3σ breach can also open a GitHub issue with a drafted intent. A model with read-only tools gets the numbers and the commits since the last quiet day and writes up what probably happened. That one costs money, so it's behind a switch.
The Intent Board
Every piece of work on this repo starts as a folder: intent/<slug>/ with an intent.md, then a spec.md, then a plan.md, committed in that order. The intent has a status: draft, accepted, in build, done or dropped.
The board reads those files straight from main and from every open pull request branch, and lays them out as a Kanban flow.
AI Practice
Most of this repo was written by Claude Code. 232 of the 283 commits on main at time of writing carry its co-author line. My job is mostly writing intents and saying no.
So the dashboard has a section for the agent itself.
Evals. A small set of tasks checks that the agent configuration steers an agent the way it's written. Does a bug report load the bugfix skill? Does an unrelated request leave it alone? Does the guard hook really stop a test edit while a fix is in progress, whichever route the agent tries?
Each task runs Claude Code on Sonnet against a throwaway copy of the repo. The verdict is a plain script that checks the files and the run's transcript. A model never grades a model.
Every task also has to prove it can fail. A free dry run plays the naive version of each task, and if the check still passes, the eval is useless.
Review. Claude Code also reviews pull requests, running as a GitHub Action. It reads each one against a written policy in REVIEW.md and the intent's plan, then leaves its findings as threads. Drafts, forks and bot PRs are skipped. Nobody needs a paid review of a dependency bump.
The card I actually look at is "Fixed after review": the share of the reviewer's threads that got resolved, either by the fix loop or by me. A reviewer whose comments get ignored isn't really reviewing anything.
It isn't free. Evals, the reviewer and the band issues all call a model, so each one sits behind a repository variable. Switch them off and the pipeline still runs. You just lose the opinions 🙂
They don't need an API key, though. They sign in with a CLAUDE_CODE_OAUTH_TOKEN from claude setup-token, which bills my Pro/Max subscription, not pay-as-you-go API credits. CI shares my usage limits.
Is It Worth It?
For a solo side project, the honest answer is probably no.
But as a playpen for a learning project then it serves its purpose.
Source: packages/testing/src/dashboard.ts.
Dashboard live here: https://newell-paul.github.io/sprint-scout/dashboard/
The rider side of the story
If you've built something like this and have advice then let me know - improvements always welcome :)
Top comments (4)
The 3-sigma gate stopping Pages publication leaves an interesting reporting edge case: the public dashboard can still show the previous green run precisely when the newest run needs attention. I'd make the displayed commit and report timestamp prominent, and retain the failed run's HTML as an artifact. Would you separate blocking a release from publishing a read-only failure report with a clear red banner? That would preserve the gate without making a stale healthy dashboard look current.
You're right about the public page. The live dashboard sits on the last green build so it can be stale on some runs.
So I could block the release, but still publish the dashboard with the red banner. I will put something on the backlog 🙂
AI assistance is most useful in CI when it produces an inspectable hypothesis, not a pass/fail authority. I’d keep the generated finding linked to the exact diff, test output, and policy that triggered it, then measure its precision by reviewer disposition. That feedback loop prevents a helpful QA assistant from becoming an expensive alert generator.
A 3-sigma rule that refuses to publish a bad day is a nice tripwire. Static HTML dashboard from the run's own reports, no scripts — clean.