I stopped re-running flaky tests: a deterministic verdict, and a Gemma that's only allowed to guess out loud

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

A test fails in CI. You press "re-run". It passes. You merge, and a week later it fails again on someone else's pull request.

I have done this far more often than I'd like to admit. Re-running is the lazy move, and it's lazy in the bad way: it throws away the evidence. The first failure, the environment, the commit and the attempt that followed it are exactly what you'd need to decide whether the test is flaky or whether you just broke something. FlakePilot is my attempt at being lazy in the good way: file the evidence once, let a fixed set of rules read it, and stop guessing.

What I Built

FlakePilot is an open-source (Apache-2.0), read-only control plane for investigating flaky tests.

• CI uploads a JUnit XML report. FlakePilot stores it with who, where and when: run id, attempt, job, commit SHA, ref, environment.
• It rebuilds each test's attempt chain (failed → passed, failed → failed, and so on).

• It classifies every failure with a versioned, deterministic rule set, ti-classification/1. The result is flaky, deterministic_regression, or an honest inconclusive such as insufficient_history. There are no confidence scores, because the data doesn't support them.
• Every verdict is stored together with the evidence it was computed from, so you can see why.
• On request, an investigation can draft a hypothesis about the cause with an open-weight model (Gemma). Code then checks that every claim cites stored evidence.

The effort it saves is the rerun-and-wonder loop. The user is whoever triages flaky CI at a small team or in an open-source project: the person who is currently reading three logs to decide whether to press "re-run".

I want to be upfront about the weight of the AI here, because the challenge asks for open-source AI at the core. In FlakePilot the verdict is never a model's output. Gemma is an optional, off-by-default advisory stage. I'll explain below why I think that's the right design, and where it leaves this submission.

Demo

• Deployed control plane: app.flakepilot.tech. It uses GitHub sign-in, so the signed-in pages need a login. The public landing page is open.
• I have no screen recording to show, so I'm not going to pretend there is one. What you can open without logging in is the landing page and the repository. The signed-in run view is what the diagrams below describe.

Please note: the execution plane is closed. FlakePilot never runs repository code, ever. It only reads reports.

Code

CodewithJha / FlakePilot

Evidence-backed flaky-test investigation: ingest CI results, classify failures, propose and verify minimal patches with human approval.

FlakePilot

Evidence-based investigation and verification of recurring test failures. FlakePilot ingests your CI test reports, reconstructs each run's attempt chains, and classifies every failure with a deterministic, versioned rule set — never a model. Since 2026-10-25 the control plane is deployed on Render at app.flakepilot.tech; the execution plane stays closed.

Pre-alpha. Not production-ready. See Status and verification for what is verified and what is not.

Contents

• What FlakePilot does
• CI reports, attempts, evidence and classification
• The investigation workflow
• Architecture
• Technology stack
• Routes and user workflows
  • GitHub authentication
  • CI ingest endpoint
  • Signed-in pages
  • Reproductions
  • Agent route (Auth0, opt-in)
• Security model and limitations
• Production deployment
• Configuration
  • GitHub App sign-in setup
  • Hosted providers (ADR-017)
• Operator-only tools
• Local development
  • Prerequisites
  • Quality gates
  • Demo journey
• Status and verification
• Documentation map
• Contributing
• Security, licence

What FlakePilot does

Flaky tests are treated like incidents, not opinions: every verdict has to come from stored, auditable CI evidence…

View on GitHub

The repository is public and Apache-2.0. The first commit is from 20 October 2026 (19:32 IST), inside the challenge window, and the history is in the repository. If any commits land after the deadline, they are noted in the README.

How I Built It

The shape of the system

The stack is Python 3.12, FastAPI and PostgreSQL (SQLAlchemy, Alembic, asyncpg), managed with uv, and checked with Ruff, strict mypy, import-linter and pytest. It is a modular monolith with bounded contexts, and the import boundaries between them are enforced by 21 import-linter contracts in CI. The UI is a small React + TypeScript app built with Vite. Python still does every authorization check and only then embeds the run's data in the page.

Fail-closed on purpose

Most of the decisions were about what the system refuses to do:

• Missing configuration disables a feature instead of guessing a default.
• A resource another tenant can see is always 404 not_found, never 403.
• A commit SHA is stored exactly as CI reported it. Its revision status stays unchecked, because FlakePilot does not verify Git ancestry.
• A test with too little history is inconclusive, not a coin flip.
• Nothing executes repository code.

Where Gemma fits

The investigation walks bounded stages: collect, research, browse, hypothesize, prioritize, brief. Each stage is skipped with a stated reason when its provider isn't configured. The hypothesize stage asks a model to suggest candidate causes from the stored evidence. It is off unless an organisation admin turns on an AI policy, and its output is stored as advisory data. It never changes a verdict and never grants anything.

The part I'm happiest with is the checking step. The model's answer has to parse as a fixed JSON shape, and each claim has to cite an evidence id that actually exists for that run. A claim that cites nothing, or cites something that isn't there, is rejected.

Why Does Open Innovation Matter?

Honestly, for this project the reason is trust and swappability, not capability.

1. The verdict doesn't depend on a model at all. That's deliberate. If the model is wrong, slow, or retired, the classification is the same. An open-weight model fits an advisory role: I can change it without touching anything that decides a verdict.
2. The model is a configuration choice. The hypothesize stage talks to either an operator-provided local endpoint or hosted Gemma. An operator who doesn't want failure logs leaving their network can point it at their own server.
3. Open code means the rules can be audited. ti-classification/1 is a plain, versioned rule set in a public repository. If you disagree with a verdict, you can read exactly why it was produced.

What I verified, and what I didn't:

• Verified live (24–25 Oct 2026): hosted Gemma 4 (gemma-4-31b-it) through the Gemini API free tier. One investigation of a seeded flaky run, on a throwaway PostgreSQL database, produced an evidence-linked hypothesis with zero unverified claims.
• Not verified: I haven't recorded a run against a local Gemma endpoint, so the "your logs never leave the machine" option is supported by the code and unit tests but not by a live run. Also, the hosted free tier may use prompts, which is a real trade-off.

So: open-weight AI is a deliberate, replaceable part of this project, but it is not the thing that makes the project work. If the challenge reads "open-source AI at the core" strictly, this is a borderline entry and I'd rather say so than dress it up.

What did not work

• Gemma's first answers all failed validation. My prompt never said that cause had to be a snake_case code or that citations had to be {"evidence_id": ...} objects, so real answers came back as invalid_hypothesis. The fix was to spell out every key and end the prompt with an example that the validator accepts. A unit test checks that example against the validator so the two can't drift apart. (Commit fd71d89.)
• A hosted call can simply fail. In one earlier end-to-end run the hypothesize stage was skipped with provider_http_error after the bounded retries. The system recorded that honestly and carried on, which is the behaviour I wanted, but it is also a reminder that a hosted free tier is not a dependency you want in the critical path.
• Several sponsor integrations are not in the product. Some were evaluated and deliberately not adopted, or exercised only against synthetic data or a local dev server. The Prize Categories section says which is which.

Current limitations

• Pre-alpha. The README says "not production-ready", and I mean it.
• The execution plane is closed, so FlakePilot can't reproduce a failure itself. It only reads what CI already recorded.
• Raw report bodies are stored as plaintext files. There is no encryption at rest yet.
• The signed-in demo needs GitHub sign-in, so a quick look from outside is limited to the landing page and the code.
• The classifier needs history. With one run, most things are honestly inconclusive.

The Lazy Payoff

I haven't measured time saved, and I'm not going to invent a number. The qualitative payoff is concrete, though: instead of pressing "re-run" and hoping, you get a stored attempt chain and a verdict that names the evidence behind it. If you want me to measure it, the honest experiment is to count how often a triage question gets answered without opening a raw log. I haven't run that yet.

What I learned

• A model that proposes while code verifies is easier to trust than a model that decides. Rejecting an unsupported claim is cheap and visible.
• "Inconclusive" is a feature. Making the system say it out loud forced me to design for missing history instead of hiding it.
• Writing the prompt's expected output down, with a validated example, solved more than any prompt tweaking did.

Next I'd like to run the hypothesize stage against a local Gemma endpoint end to end and record it, add a no-login demo, and measure the triage claim above.

Prize Categories

FlakePilot uses each of these partners for one specific job. Here is what each one does in the project.

• Best Use of Gemma: Gemma 4, served through the Gemini API, powers the optional hypothesize stage of an investigation. It reads the stored evidence for a failing test and proposes candidate causes. Every claim it makes must cite an evidence id that exists for that run, and code rejects anything that does not. The output is stored as advisory data and never changes a verdict.
• .tech domain: flakepilot.tech is the project's domain, and app.flakepilot.tech serves the control plane. GitHub sign-in, the run view and the API all run on it with a valid certificate.
• Render: the API, the background worker and the PostgreSQL database run on Render. A digest-pinned image is deployed after CI passes, database migrations run before each release, and the database has no public connections.
• SerpApi: the research stage of an investigation runs one Google search built from the failing test names. It keeps deduplicated https sources only, within a fixed budget, and records them as evidence for the investigation.
• ElevenLabs: on explicit request, FlakePilot turns an investigation into a short spoken brief. The brief is made of counts and the report link only, with no repository text, and is returned as an MP3 that is not stored.
• Auth0 for AI Agents: an agent can investigate a run on someone's behalf through a delegated token. FlakePilot verifies the token against the Auth0 tenant's keys, and a token can only narrow what the person can do, never widen it. An org admin binds and unbinds agents, bindings expire within eight hours, and every action is attributed to the agent.
• MongoDB Atlas: classified run reports can be exported to Atlas. The export is write-only and idempotent, and each document holds ids, counts and verdicts with their evidence, with no test names or artifact keys. PostgreSQL stays the system of record.
• Tiger Data: the reliability analytics run on Tiger Cloud's PostgreSQL service. They compute per-test failure history, per-environment counts, failure clusters, and day and week trends of failures and flakiness compared with the previous window.
• TabPFN: TabPFN predicts which failing tests to look at first. FlakePilot sends it only numeric history rows with labels, and an evaluation harness checks it against the deterministic baseline with a temporal split and past-only features.
• Temporal: an investigation can run as a Temporal workflow. Retries happen only for transient failures, cancellation propagates, and if a worker dies mid-run, a replacement finishes the job and exactly one investigation is recorded.
• Sentry: an error reporter sends allow-listed events (exception class, frames, a message template, request id and route) from the API process. It never sends repository text or secrets.
• Mastra: the investigation runs as a bounded workflow of collect, research, browse, hypothesize and prioritize stages, and records one provenance row at the end. A Mastra workflow built on @mastra/core calls the same agent route as an authority-free client.
• Browserbase: the browse stage captures allowlisted research pages through a Browserbase adapter. Only exact public hosts are allowed, redirects are refused, and each capture is stored as hashed evidence whose text never reaches the model.

Comments

[Parsa Mohammadi]

Gemma having to cite an evidence id or get thrown out is a really good rule. one thing worth watching is a test that’s been flaky for weeks and then turns into deterministic_regression, that’s usually a real break hiding behind a name everyone ignores. does it keep enough history to catch that flip?

[Adam Lewis]

Your admission that the hosted free tier may use prompts matters, because that trade-off decides whether the local endpoint option ever gets used. Running the validator against those cases would settle it. If the hypothesize stage cites stored evidence and code checks it exists, does that check only confirm the reference is real, or does it also confirm the claim follows from it? Those are different guarantees and the article leans on the first while reading as if it has the second. If the validator only checks existence, a model can cite the right id and still draw the wrong cause from it.