RepoSage: Local-First GitHub Audits with Open-Weight Gemma 4 & Anti-Hallucination Gates

Salman Ahmad — Tue, 20 Oct 2026 20:25:55 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Most software developers genuinely want thorough code reviews, security governance, and CI/CD audits. But in practice, comprehensive repository hygiene is one of the most procrastinated, dreaded tasks in engineering. We open a repository, dread the prospect of manually inspecting dozens of YAML workflow files, auditing transitive dependencies, verifying license files, checking branch protection settings, and writing out tickets. So we put it off until an incident occurs.

Existing AI linters and code review bots promise to automate this, but introduce two severe dealbreakers:

The Credential Dilemma: They ask you to paste a sensitive Personal Access Token (PAT) into a web app or .env file, creating an unnecessary attack surface.
The Hallucination Trap: General-purpose LLMs routinely invent phantom files, cite workflows that don't exist, or claim a security policy is missing when it sits right in .github/SECURITY.md.
Privacy Exposure: Auditing proprietary code with closed-source cloud APIs forces organizations to transmit intellectual property across the public internet.

In the spirit of International Sloth Day and this week's "Slow and Steady" theme, I built RepoSage — a tool that channels your inner sloth by doing the tedious, heavy-lifting repository auditing for you, while maintaining rigorous, deterministic standards.

RepoSage is a local-first, evidence-grounded GitHub repository auditor powered by open-weight Gemma 4 models. It connects natively to your machine's authenticated GitHub CLI (gh) session with zero token pasting, computes an instant deterministic 0–100 Health Score across 6 dimensions before calling any AI, cross-verifies every AI claim against your actual Git tree to quarantine hallucinations, and converts verified findings into real GitHub Issues or turnkey AI fix prompts in a single click.

Key Features

🔑 Zero-PAT Friction: Bridges directly to your local gh auth token in-memory. Zero tokens saved, logged, or sent to the browser.
📐 Deterministic Health Baseline: Computes a reproducible 0–100 Health Score across 22 rules (CI/CD security, test coverage, dependency hygiene, documentation, community health, and velocity) completely offline before any AI runs.
🧠 Dual-Runtime Open-Weight AI (Gemma 4):
- Local Privacy Mode: Runs Gemma 4 E4B locally on Ollama (/api/chat with structured JSON schema) for 100% air-gapped, zero-cloud confidentiality.
- Hosted Mode: Runs Gemma 4 26B (MoE) via Google AI Studio for deep architectural synthesis.
🛡️ Anti-Hallucination Citation Verifier: Every file path and check cited by the model is validated against the repository's recursive Git tree. Any hallucinated file or nonexistent check is immediately quarantined as UNVERIFIED_PATH or UNVERIFIED_CHECK and stripped from the action plan.
⚖️ Fair Comparison Arena: Generates a canonical SHA-256 hashed evidence manifest ensuring both local and hosted Gemma runtimes receive byte-identical evidence for benchmark comparisons.
⚡ 1-Click GitHub Remediation: Converts verified findings into pre-formatted, actionable GitHub Issues through the CLI bridge.
🛠️ Turnkey AI Fix Prompts: Generates structured prompts tailored for Claude Code, Cursor Composer (@file), Devin, and Aider CLI (/add) to remediate vulnerabilities and achieve a 100/100 score automatically.

Demo

Here is how RepoSage orchestrates an audit from evidence collection to remediation:

flowchart TD
    A[Local Developer Machine] -->|gh auth token| B[GitHub CLI Bridge]
    B -->|GraphQL & REST APIs| C[Evidence Collector]
    C -->|Recursive Git Tree & Configs| D[Deterministic Rule Engine]
    D -->|Instant 0-100 Score| E[Canonical Manifest Compiler]
    E -->|Secret Redaction & SHA-256 Hash| F{Gemma 4 Runtime}
    F -->|Local Mode: Ollama| G[Gemma 4 E4B]
    F -->|Hosted Mode: AI Studio| H[Gemma 4 26B MoE]
    G --> I[Structured JSON Output]
    H --> I
    I --> J[Anti-Hallucination Citation Verifier]
    J -->|Verified Citations| K[Interactive Audit Dashboard]
    J -->|Hallucinated Citations| L[Quarantine Telemetry]
    K --> M[1-Click GitHub Issue Creation]
    K --> N[Turnkey AI Agent Fix Prompts]

The User Flow

Instant Session Detection: Opening RepoSage at http://localhost:3000 immediately surfaces your public and private repositories using your authenticated gh session. No login screens or PAT inputs.
Deterministic Scan: Selecting a repository kicks off the audit. In less than 500ms, the deterministic engine scans workflow permissions, lockfile integrity, license metadata, and branch policies to set a baseline score.
Streaming AI Reasoning: Gemma 4 streams structured risk assessments and architectural remediation advice over Server-Sent Events (SSE).
Citation Validation: The Citation Verifier inspects every claim. Notice how findings display a verified badge linked directly to the file in the repository.
Remediation: Click "Create GitHub Issue" to file the issue, or "Copy Fix Prompt" to hand the task over to an AI coding agent.

Code

ahmmikun / HacktoberFest-x-Fluxxion

Official Open-Source AI Project for MLH Hacktoberfest Hack Day Lahore x Fluxxion 2026

RepoSage 🔎

Local-first, evidence-grounded GitHub repository auditing powered by open-weight Gemma 4.
Official open-source AI submission for MLH Hacktoberfest Hack Day Lahore × Fluxxion 2026, targeting the Best Open-Source AI Project challenge.

Note

Project status: v1.0.0 implemented. The full audit pipeline, API, and dashboard are in the repository. npm ci, npm run typecheck, and npm run build pass on Node.js 24.22 with npm 11.19 (Windows). There is no automated test suite or CI workflow yet; see Known Limitations.

Overview

RepoSage audits GitHub repositories for CI/CD security, testing, dependency hygiene, documentation, security governance, and…

View on GitHub

RepoSage is organized as a clean TypeScript npm-workspaces monorepo:

packages/contracts: Shared Zod schemas, strict TypeScript contracts (LlmAuditOutputSchema, AuditReport, Finding), and JSON Schema converters.
apps/api: Node.js/Express server orchestrating the audit pipeline:
- src/services/gh-cli-bridge.ts: In-memory gh CLI session bridge with 5-minute cache and strict shell-injection protection.
- src/services/deterministic-engine.ts: 22-rule deterministic scoring engine across 6 dimensions.
- src/services/manifest-compiler.ts: Evidence compiler with secret redaction and canonical SHA-256 serialization.
- src/services/gemma-service.ts: Dual-runtime model harness for Ollama (gemma4:e4b) and Google AI Studio (gemma-4-26b-a4b-it).
- src/services/citation-verifier.ts: Anti-hallucination engine that cross-verifies model findings against the real Git tree.
- src/services/consensus-service.ts: Computes agreement and latency matrices between local and hosted runtimes.
apps/web: Next.js 15 (App Router) frontend with Tailwind CSS v4, real-time Server-Sent Events (SSE) streaming, and Mission Control dashboard.

How I Built It

1. Dual-Runtime Open-Weight AI Architecture

Rather than relying on proprietary APIs, RepoSage is engineered specifically around open-weight Gemma 4:

Runtime	Model	Engine	Purpose
Local Privacy Mode	`gemma4:e4b`	Ollama `/api/chat` with JSON schema	100% offline, air-gapped code review with zero network egress
Hosted Mode	`gemma-4-26b-a4b-it`	Google AI Studio via `@google/genai`	Deep architectural reasoning and edge-case synthesis

Both models are strictly constrained to a JSON Schema derived directly from our Zod contract (LlmAuditOutputSchema) with temperature: 0.1. If an inference response violates schema constraints, an automated single-retry repair loop feeds the parsing error back to the model to guarantee contract validity.

2. The Anti-Hallucination Citation Verifier

LLM code review tools are notorious for inventing non-existent file paths (e.g., claiming .github/workflows/security.yml is vulnerable when the file doesn't exist).

To solve this, RepoSage implements a two-gate verification pipeline:

The Git Tree Gate: The evidence collector gathers the repository's complete recursive Git tree via GraphQL. The Citation Verifier checks every filePath referenced in a finding against the true set of repository paths.
The Deterministic Truth Gate: If the model references a deterministic check ID (e.g., CICD_PERMISSIONS), the verifier confirms that the check was executed and actually flagged an issue.

// Core Citation Verifier Logic
export function verifyFinding(
  finding: RawFinding,
  validPaths: Set<string>,
  failedCheckIds: Set<string>
): VerifiedFinding {
  if (finding.filePath && !validPaths.has(finding.filePath)) {
    return { ...finding, status: "UNVERIFIED_PATH", quarantined: true };
  }
  if (finding.checkId && !failedCheckIds.has(finding.checkId)) {
    return { ...finding, status: "UNVERIFIED_CHECK", quarantined: true };
  }
  return { ...finding, status: "VERIFIED", quarantined: false };
}

Any finding flagged as unverified is quarantined into telemetry, excluded from the actionable plan, and barred from issue creation.

3. What Went Wrong & Lessons Learned

Token Redaction in Manifests: Early in development, repos with .env.example files containing dummy keys or base64 tokens occasionally caused models to fixate on the example strings. We implemented a regex-based secret and key scrubber that redacts AWS keys, GitHub PATs, JWTs, and bearer tokens before canonical manifest hashing.
Shell Injection Prevention: Because we bridge with the gh CLI, user-supplied repository names (owner/repo) could theoretically introduce shell injection vulnerabilities. We replaced all generic shell execution with child_process.execFile, passing arguments strictly as string arrays and validating repository identifiers against /^[a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+$/.
Context Window Budgets: Large repositories with thousands of files can overwhelm local context windows. We capped file trees to high-signal config files (.github/**, package.json, tsconfig.json, Dockerfile, LICENSE, etc.) while retaining top-level file names to keep the prompt compact and fast.

Why Does Open Innovation Matter?

Air-Gapped Confidentiality for Private IP: Most proprietary codebases cannot be shared with third-party cloud APIs due to compliance (SOC2, HIPAA, GDPR) and privacy restrictions. By running open-weight Gemma 4 locally via Ollama, RepoSage enables developers to audit sensitive internal infrastructure and proprietary algorithms completely offline.
Predictable Zero-Cost Operations: Cloud API subscriptions for code analysis scale with token counts and team seats. Open-weight models run on hardware you already own without recurring SaaS invoices.
Customizability & Freedom from Deprecation: When building on open weights, your audit pipeline isn't vulnerable to sudden API deprecations, pricing changes, or surprise model shifts. You can fine-tune, quantize, and pin specific model versions at will.
Slow and Steady Wins the Race: In automated code audits, you don't need real-time chat latency. A small, efficient open-weight model like Gemma 4 E4B running steadily on consumer hardware produces reliable, structured findings while consuming minimal energy.

The Lazy Payoff

In honor of International Sloth Day, here is the concrete payoff for the developer who would rather let AI handle the heavy lifting:

The Manual Way: Auditing a typical repository requires 1.5 to 3 hours of manually inspecting GitHub Actions workflows, checking secret permissions, verifying lockfiles, auditing documentation compliance, and writing out detailed GitHub issues.
The RepoSage Way:
- Click 1: Select repository.
- 500ms: Deterministic engine computes 0–100 health baseline across 22 rules.
- 15–25s: Local Gemma 4 interprets risks, highlights root causes, and drafts remediation plans.
- 0ms: Citation Verifier quarantines hallucinations automatically.
- Click 2: Click "Create GitHub Issue" to file the issue, or "Copy Fix Prompt" to paste directly into Cursor Composer or Claude Code.
Total Time Saved: Over 95% reduction in manual audit toil, letting you relax while your repositories stay secure.

Prize Categories

Featured Category: Best Use of Gemma ($200) RepoSage is built natively for Google's open-weight Gemma 4 family, leveraging Gemma 4 E4B for local zero-cloud auditing and Gemma 4 26B for hosted analysis, with structured Zod output validation, automatic repair loops, and dual-runtime benchmark comparisons.
Also Built For DigitalOcean: RepoSage is fully containerized with Docker and ready for 1-click deployment on DigitalOcean Droplets and App Platform.

New to this Community

Salman Ahmad — Wed, 24 Dec 2025 08:48:28 +0000

I am exited to be the part of it.

DEV Community: Salman Ahmad