DEV Community: Alexander Korovin

What does a green CI check actually prove?

Alexander Korovin — Thu, 10 Sep 2026 11:25:25 +0000

When I look at a pull request, I do not really care that a job named test is green. I care that the right tests ran, on the right commit, under a policy the pull request could not quietly weaken.

The green badge is still useful. It compresses a lot of work into one bit: pass or fail. Trouble starts when that bit becomes the whole argument for merging.

GitHub can associate a check run with a commit SHA, and branch protection can require a check from a selected GitHub App. But GitHub also documents an important boundary: required status checks are selected by name and do not take the workflow, matrix, or event type into account. A check name is therefore a useful merge control, not a complete answer to four separate questions:

Which exact candidate produced this result?
Which workflow and concrete job produced it?
Did the executed checks cover every file changed by this candidate?
Which trusted policy decided that the evidence was enough?

You can ignore much of this in a small repository with one obvious workflow. It becomes important once a repository has several workflows, path-specific checks, reruns, generated code, or automation that copies CI results into another gate.

Reproduce the missing binding

I wanted to see this failure in a tiny example—not just describe it—without needing credentials or access to a real repository. So I added a synthetic pattern named FFA-001 to Fleet Failure Atlas. It models a deliberately weak gate that accepts a successful receipt by status while forgetting to bind the receipt to the current candidate.

git clone --depth 1 https://github.com/korovin-aa97/fleet-failure-atlas.git
cd fleet-failure-atlas
python3 atlas.py run FFA-001 --mode reproduce
python3 atlas.py run FFA-001 --mode detect
python3 atlas.py run FFA-001 --mode regress

The relevant output is small:

reproduce: vulnerable_gate_accepts = true
           receipt_sha = 1111...1111
           candidate_sha = 2222...2222

detect:    head_sha_mismatch
           coverage_not_bound_to_head

regress:   repaired_gate_accepts = false
           fresh_receipt_accepts = true

One easy detail to miss: the command exits successfully because the fixture proved its expected condition. In reproduce mode, “pass” means the contained failure was reproduced—not that the vulnerable predicate is safe.

The fixture uses synthetic 40-character identities. It is not evidence that GitHub attached a check run to the wrong commit, and it is not presented as a real incident. It demonstrates a more general engineering error: downstream automation accepted a detached green result without checking its subject and coverage.

What the evidence contract needs

A stronger decision has to join several facts instead of looking at a display name and conclusion in isolation.

1. Subject identity

The evidence must name the exact pull-request head SHA being evaluated. A receipt for yesterday’s commit is irrelevant even when every test in that receipt passed.

2. Producer identity

The expected GitHub App is useful, but a multi-workflow repository often needs more: workflow path, event type, concrete job, workflow run, and latest attempt. This prevents an identically named job in another workflow from being mistaken for the required producer.

GitHub itself recommends unique job names across workflows because duplicate names can create ambiguous required-check results.

3. Changed-surface coverage

“Tests passed” is incomplete when no rule connects changed paths to the tests that should have run. A policy can map surfaces to evidence:

[[surfaces]]
name = "python"
patterns = ["src/**/*.py", "tests/**/*.py", "pyproject.toml"]
checks = ["test"]

The safest default is fail-closed: an unmapped changed file is a finding, not a silent exemption.

4. Policy provenance

If a pull request can weaken the policy that judges the same pull request, the result is circular. Load the manifest from the base commit, protect the manifest and verifier paths, and keep the workflow that invokes the judge under an independent control.

Base-held policy answers “what policy applies?” A protected required workflow or independent review answers “who ensures the judge runs?” These are different parts of the boundary.

5. Freshness and reruns

An older successful attempt should not hide a newer failure or an in-progress rerun. Evaluate the newest unambiguous concrete attempt, validate timestamps, and treat retrieval ambiguity as an invalid evaluation instead of guessing.

Turn the decision into a receipt

I implemented the same model in CI Evidence Gate, a read-only GitHub Action. Its local demo creates disposable Git repositories and returns three verdicts:

git clone --depth 1 https://github.com/korovin-aa97/ci-evidence-gate.git
cd ci-evidence-gate
PYTHONPATH=src python3 scripts/run_demo.py

valid        sufficient   findings=none
failed test  insufficient findings=required-check-conclusion
policy edit  invalid      findings=protected-policy-change

The three outcomes intentionally separate two kinds of failure:

Verdict	Meaning
`sufficient`	The declared evidence exists for the exact subject and covers the changed surface.
`insufficient`	The evaluation is trustworthy, but required evidence is missing, stale, incomplete, or unsuccessful.
`invalid`	The judge cannot trust its policy, inputs, provenance, or retrieval result.

The receipt records the base and head SHA, base-policy digest, changed files, matched surfaces, expected checks, observed producer metadata, findings, and the final verdict. That makes the merge decision inspectable after the green or red UI indicator is gone.

The production Action requests only contents: read, checks: read, and actions: read. It does not write comments, checks, pull requests, repository files, or artifacts. The local demo uses an in-memory provider; production evaluation queries GitHub rather than accepting candidate-supplied evidence.

What this still does not prove

An evidence contract is deliberately narrower than a correctness claim. Even a perfectly bound receipt does not prove that:

the tests assert the right behavior;
the source code is correct;
an evidence-producing job did meaningful work instead of a no-op;
a compromised runner or expected GitHub App is trustworthy;
the repository rules actually prevent bypass;
a candidate-controlled workflow cannot stop the gate from starting.

For a small repository with one obvious workflow, native branch protection may already be enough. Adding another gate creates maintenance and availability cost, so the extra machinery should correspond to a real provenance or coverage problem.

A practical review checklist

Before introducing a custom gate, I would review the existing CI in this order:

Give jobs unique names across workflows.
Require the expected source App where GitHub supports it.
Confirm every decision is tied to the exact candidate SHA.
Decide whether workflow path and event type matter for this repository.
Map changed surfaces to required checks and fail on unmapped files.
Load policy from a trusted base and protect the verifier’s own files.
Evaluate the latest concrete rerun rather than any historical success.
Fail closed on ambiguous provenance or incomplete API results.
Pin third-party Actions to full commit SHAs.
Keep the claim precise: evidence sufficiency is not program correctness.

The question I now use in review is not just “is CI green?” It is “can I explain why this green result belongs to this change?” If the merge is important enough to gate, the evidence behind that verdict should survive the color of the UI.

Runnable references:

Disclosure: I maintain both open-source projects linked above.

I stopped writing bug reports. Now I complain at my screen.

Alexander Korovin — Thu, 30 Jul 2026 12:35:05 +0000

I build a product solo. That also makes me the QA team.

Finding bugs was never the slow part. The slow part came after. I would reproduce a bug, and then spend the next hour turning it into a ticket: watch the recording again, write down the steps, scroll for the frame where it breaks, copy the error text, and try to match "it froze right here" with the right lines in the logs.

At some point I noticed a stupid thing. While reproducing a bug, I talk. I say things like "why is this button doing nothing" and "wait, the price is wrong again". Everything a good bug report needs was already in that moment. I just had no way to keep it.

So I built one.

The workflow now

I press record, click through the app, and complain out loud. Then I open Claude Code and run one command:

/talkthrough:bug recordings/checkout-bug.mp4

Behind it is talkthrough-mcp, a free open source MCP server (MIT) that runs on my machine. What happens:

The recording gets indexed locally: a transcript with timestamps, key frames, OCR text from the screen, and the real time of day for each moment when the file metadata allows it.
Claude does not read the whole video. It queries that index: search the words, pull the exact moment, look at the actual frame.
It writes one GitHub issue draft: what happened, what I expected, steps to reproduce, severity. Every claim has proof attached: my words from the recording, the timestamp, the error text from the screen, and the frame reference.

Here is a full 78-second run, not edited. A narrated recording goes in, a ready-to-file bug-report.md comes out:

The draft is the end of the pipeline on purpose. /talkthrough:bug never creates or edits an issue on its own. I read the draft, sometimes fix a detail, then file it with gh or paste it into the tracker. An agent that silently writes into your bug tracker is a bigger problem than one extra step.

Why everything runs locally

talkthrough never uploads the recording. Whisper runs on CPU, OCR and frame extraction are local, and there is no talkthrough account, no cloud backend, no telemetry, no extra API key. Claude sees only the text and images that the MCP tools return, inside your normal Claude Code session.

This matters more than it sounds. Recordings are full of things you do not want to send to a random service: customer data on the screen, internal URLs, half-finished features. And meetings are even worse. Which brings me to the part I did not plan.

The use case my colleagues found

I shared the tool with colleagues at my day job. They mostly ignored the bug part and pointed it at recordings of real work meetings.

Since v0.2 talkthrough has local speaker diarization (sherpa-onnx: no torch, no accounts, no GPU). A recorded meeting becomes a transcript where speakers are labeled S1, S2 and so on, and every remark carries the real time of day. So you can ask "what did we decide about the migration" and get an answer with a quote, a speaker and a timestamp, without rewatching anything.

One real number from their feedback: a 65-minute Teams call with about 16 people took around 15 minutes to process on a busy work laptop, diarization included. Not instant. Still much better than an hour of rewatching.

Their reports also shaped the tool more than my own plans did. For example, the warning you get when the detected speaker count looks implausible came straight from a real meeting where clustering went wrong. When your only test data is your own voice, real users are a gift.

If you want to check the numbers

I do not like demo magic, so the repo has a fully reproducible example: examples/bug-from-silent-recording. It is a Playwright-recorded clip with no audio track at all (silent recordings work too, through frames and OCR), plus the unedited agent output. The job id is the sha256 prefix of the file, so you can verify every number on your machine: same file, same id, same keyframes.

Honest limits

Local files only. No YouTube, no URL ingest (planned).
The first run downloads the models, so it is slow once.
The analysis uses the transcript plus key frames, so it can miss motion between frames. You can still pull an exact full-resolution frame for any timestamp.
Diarization quality depends on the audio. Very short remarks sometimes get the wrong speaker; passing num_speakers when you know the headcount helps a lot.

Try it

In Claude Code:

/plugin marketplace add korovin-aa97/talkthrough-mcp
/plugin install talkthrough@talkthrough

It is a plain stdio MCP server, so it is not tied to Claude: configs for Cursor, Codex CLI and ten more clients are in the repo. Everything is here: https://github.com/korovin-aa97/talkthrough-mcp

One design question I keep going back and forth on: should /talkthrough:bug stay draft-only, or should it also be able to create the GitHub, Linear or Jira issue after an explicit confirmation? I would honestly like to know what you would want.