DEV Community: Finley Sun

A Reusable Smoke-Test Harness for Newly Released Open Models (Before You Bet a Project on One)

Finley Sun — Mon, 10 Aug 2026 10:34:50 +0000

Every few weeks a new open-weights model drops — the recent wave of releases from teams like MiniMax is a good example — and my timeline fills with benchmark screenshots. The problem: those benchmarks rarely answer the question I actually have, which is "does this model survive the boring tasks in my repo?"

So instead of reading another leaderboard, I built a small, reusable smoke-test harness. It runs the same fixed set of prompts against any OpenAI-compatible endpoint, scores the outputs with cheap deterministic checks, and writes a report I can diff across models. When a new model is announced, I can have a first honest signal in under half an hour — often using free tiers and free hosted compute, so the experiment costs nothing but time.

This post is the harness, the reasoning behind it, and its limits.

Why not just trust the published benchmarks?

Published benchmarks measure aggregate capability. My risks are narrower and weirder:

Does the model follow my project's formatting conventions (e.g., type hints, no Any)?
Does it hallucinate imports that don't exist?
Does it stay coherent on the second turn, when I ask it to modify its own output?
Is latency acceptable for interactive use, not just batch jobs?

A 20-prompt harness tuned to my actual work answers these better than any public score. And because it's just HTTP calls, it works against any provider that exposes an OpenAI-compatible API — which is now most of them, including the open-weight releases everyone is currently talking about.

Where to run it for free

Two things usually block this kind of experiment: API cost and a place to run the harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode happens to remove both blockers for this workflow — it offers free access to a selection of models and a free server option, so you can point the harness at a model you're curious about and run the whole evaluation on hosted compute without a local GPU or a credit card. That's the entire role it plays here; the harness itself works against any compatible endpoint, and you should treat free availability as something that can change, not a guarantee.

There's also a point worth making about the open source spirit here. The reason a harness like this is even possible is that providers converged on open, documented API shapes, and that teams releasing open weights — MiniMax and others — let anyone inspect, host, and poke at their models instead of hiding them behind a black box. MonkeyCode's approach fits the same pattern: free access lowers the barrier to verifying things yourself rather than trusting marketing. Open ecosystems compound because they let outsiders test claims. This harness is my small contribution to that loop — take it, run it against whatever model just launched, and publish your own results.

The harness

The design constraints:

Fixed prompt set, versioned. Prompts live in a JSON file so results are comparable across models and across time.
Deterministic scoring where possible. Compile checks, regex checks, import existence — things that don't require a judge model.
Raw outputs saved. So a human (me) can re-review borderline cases later.

prompts.json — a minimal starter set:

[
  {
    "id": "py_parse_logs",
    "kind": "python_function",
    "prompt": "Write a Python function `parse_durations(lines: list[str]) -> dict[str, float]` that reads log lines like 'task=backup duration=1.5s' and returns total seconds per task. Include type hints. No third-party imports.",
    "entrypoint": "parse_durations",
    "test": "result = parse_durations(['task=a duration=1.5s', 'task=a duration=0.5s', 'task=b duration=2s']); assert result == {'a': 2.0, 'b': 2.0}, result"
  },
  {
    "id": "py_fix_own_code",
    "kind": "python_followup",
    "prompt": "Now change the function so invalid lines are skipped instead of raising. Return only the updated function.",
    "entrypoint": "parse_durations",
    "test": "result = parse_durations(['garbage line', 'task=a duration=1s']); assert result == {'a': 1.0}, result"
  }
]

smoke.py:

"""Smoke-test an OpenAI-compatible model endpoint against a fixed prompt set.

Usage:
    export SMOKE_BASE_URL="https://your-endpoint/v1"
    export SMOKE_API_KEY="..."
    export SMOKE_MODEL="model-name-as-served"
    python smoke.py prompts.json report_dir/
"""
import json
import re
import sys
import time
import urllib.request
from pathlib import Path


def chat(messages: list[dict], base_url: str, api_key: str, model: str) -> tuple[str, float]:
    body = json.dumps({
        "model": model,
        "messages": messages,
        "temperature": 0.0,
    }).encode()
    req = urllib.request.Request(
        f"{base_url.rstrip('/')}/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}",
        },
    )
    start = time.monotonic()
    with urllib.request.urlopen(req, timeout=120) as resp:
        payload = json.loads(resp.read())
    elapsed = time.monotonic() - start
    return payload["choices"][0]["message"]["content"], elapsed


def extract_code(text: str) -> str:
    """Pull the first fenced code block, or fall back to the raw text."""
    m = re.search(r"```

(?:python)?\n(.*?)

```", text, re.DOTALL)
    return m.group(1) if m else text


def run_python_check(code: str, entrypoint: str, test: str) -> tuple[bool, str]:
    """Execute generated code plus its assertion in a throwaway namespace.

    NOTE: this executes model output. Run in a disposable environment
    (container, free hosted server, VM) — never on a machine with secrets.
    """
    namespace: dict = {}
    try:
        exec(compile(code, "<generated>", "exec"), namespace)
        assert callable(namespace.get(entrypoint)), f"{entrypoint} not defined"
        exec(compile(test, "<test>", "exec"), namespace)
        return True, ""
    except Exception as exc:  # noqa: BLE001 - we want the failure text in the report
        return False, f"{type(exc).__name__}: {exc}"


def main() -> None:
    import os

    prompts_path, report_dir = Path(sys.argv[1]), Path(sys.argv[2])
    report_dir.mkdir(parents=True, exist_ok=True)
    base_url = os.environ["SMOKE_BASE_URL"]
    api_key = os.environ["SMOKE_API_KEY"]
    model = os.environ["SMOKE_MODEL"]

    prompts = json.loads(prompts_path.read_text())
    results = []
    history: list[dict] = []

    for item in prompts:
        if item["kind"] != "python_followup":
            history = []  # fresh conversation per task chain
        history.append({"role": "user", "content": item["prompt"]})
        try:
            reply, latency = chat(history, base_url, api_key, model)
        except Exception as exc:  # noqa: BLE001
            results.append({"id": item["id"], "passed": False, "error": str(exc)})
            continue
        history.append({"role": "assistant", "content": reply})

        code = extract_code(reply)
        passed, detail = run_python_check(code, item["entrypoint"], item["test"])
        (report_dir / f"{item['id']}.py").write_text(code)
        results.append({
            "id": item["id"],
            "passed": passed,
            "latency_s": round(latency, 2),
            "detail": detail,
        })

    passed_count = sum(r["passed"] for r in results)
    report = {
        "model": model,
        "passed": f"{passed_count}/{len(results)}",
        "results": results,
    }
    (report_dir / "report.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()

The whole thing is stdlib-only on purpose: no dependencies means it runs anywhere, including a freshly provisioned free server, with zero setup beyond three environment variables.

How I read the results

The pass count is the least interesting number. What I actually look at:

Failure mode. A syntax error in extracted code suggests formatting problems; a wrong assertion result suggests reasoning problems; a missing entrypoint suggests instruction-following problems. Different failures disqualify a model for different jobs.
Follow-up behavior. The py_fix_own_code prompt is where many models quietly degrade — they rewrite the whole thing and break the original behavior, or ignore "return only the updated function."
Latency distribution. One slow response is noise; consistently slow responses rule out interactive use.
The saved .py files. I skim every one. Deterministic checks can't catch code that passes the test but is ugly, fragile, or full of phantom imports elsewhere.

Because each run writes into its own report_dir/, comparing two models — or the same model two months apart — is just diff -ru.

Limitations, and who shouldn't do this

Twenty prompts is not an evaluation. It's a smoke test. It can disqualify a model for your workflow; it cannot rank models in general. If you need rigorous comparison, look at proper eval frameworks and far larger task sets.
Executing generated code is dangerous. The harness runs whatever the model emits. Only run it in an environment with no secrets, no network access to internal systems, and no persistent disk you care about. A disposable free server instance is actually a reasonable fit here for exactly this reason.
temperature=0.0 is not fully deterministic on most serving stacks. Rerun before drawing strong conclusions from a one-run difference.
Free tiers change. Don't build CI that hard-depends on any provider's free model access or free server remaining available; treat them as experiment infrastructure, not production infrastructure. Check the provider's current terms before relying on either.
If your tasks need judgment-based scoring (design reviews, prose quality), deterministic checks won't get you far — you'd need a judge model or human review, which is a bigger project.

Try it on the next launch

The next time an open model drops and the hot takes start flying, clone this harness, write five prompts from your actual current project, and run it before forming an opinion. Half the value is the harness; the other half is the habit of verifying claims yourself — which is, at its core, what the open source ecosystem is supposed to make easy. If you want a zero-cost sandbox for that first run, MonkeyCode's free model access and free server option are one way to get both the endpoint and the disposable environment in the same place — but the harness doesn't care where you point it.

A Local AI Review Pass for Python Diffs Before You Open the PR

Finley Sun — Mon, 10 Aug 2026 08:08:48 +0000

My last post here was about a pre-push gate that fails loud instead of often. Since then I've been running an experiment on top of it: adding an AI review pass to the same gate, so a second set of (artificial) eyes looks at my diff before any human does.

The constraint I set for myself: it had to cost nothing to run, it had to work on plain git diff output, and it had to be a gate — meaning it produces a decision, not a wall of commentary I ignore.

Here's the workflow and the script. Everything below runs locally against a diff; the only network call is the model inference.

The problem with AI review as usually practiced

Most AI-assisted review I've seen fails in one of two ways:

It's a chat, not a gate. You paste code into a chat window, get prose back, and nothing forces a decision. The signal-to-noise ratio decays fast.
It reviews files, not diffs. Reviewing a whole file with a model means it flags pre-existing issues you didn't touch, and you learn to ignore it.

The fix for both: review only the diff, require structured output, and make the script exit non-zero above a severity threshold. Same philosophy as a linter with --max-warnings.

Why free models are good enough for this

A diff-review gate is a high-volume, low-stakes-per-call workload. You run it on every push, most pushes are fine, and a false positive costs you thirty seconds of reading. That's exactly the profile where a free model tier makes sense: you'd never justify a paid API call for "check whether my 40-line diff does something dumb," but you'll happily run it if it costs nothing.

I'm using MonkeyCode for this, which offers free model access and a free server option, so the whole gate runs without a billing account or a GPU on my machine. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The script below is deliberately written against a generic OpenAI-compatible chat endpoint, though — if you have another provider with an OpenAI-style API, change the base URL and it works the same.

One thing I want to be honest about up front: I haven't benchmarked the free tier's latency or rate limits under heavy use, and I don't know what its long-term availability terms are. For a pre-push hook that makes one call per push, none of that has mattered to me so far. If you're gating CI for a team of fifty, you'd want to verify those limits first.

The gate script

Save as ai_review_gate.py, run it from your pre-push hook or manually with python ai_review_gate.py origin/main.

#!/usr/bin/env python3
"""AI review gate for Python diffs.

Reviews only the current diff (not whole files), requires structured
JSON findings, and exits 1 if any finding meets the severity threshold.
"""
import json
import os
import subprocess
import sys

import urllib.request

BASE_URL = os.environ.get("AI_REVIEW_BASE_URL", "https://YOUR-ENDPOINT/v1")
API_KEY = os.environ.get("AI_REVIEW_API_KEY", "")
MODEL = os.environ.get("AI_REVIEW_MODEL", "default")  # whatever free model your endpoint exposes
BLOCK_AT = {"high", "critical"}  # severities that fail the gate
MAX_DIFF_CHARS = 24_000

PROMPT = """You are reviewing a git diff of a Python project.
Report ONLY issues introduced or worsened by this diff. Do not comment
on pre-existing code the diff merely touches.

For each issue, return a JSON array element:
  {"file": str, "line": int|null, "severity": "low"|"medium"|"high"|"critical",
   "kind": "bug"|"security"|"correctness"|"regression-risk",
   "summary": str (one sentence), "suggestion": str (one sentence)}

Return [] if the diff looks fine. Return ONLY the JSON array, no prose.

DIFF:
"""

def get_diff(base: str) -> str:
    diff = subprocess.run(
        ["git", "diff", f"{base}...HEAD", "--", "*.py"],
        capture_output=True, text=True, check=True,
    ).stdout
    if len(diff) > MAX_DIFF_CHARS:
        # Truncate rather than skip silently — a truncated review is
        # still a review; a silent pass is not.
        diff = diff[:MAX_DIFF_CHARS] + "\n# [diff truncated for review]\n"
    return diff

def call_model(diff: str) -> list[dict]:
    body = json.dumps({
        "model": MODEL,
        "messages": [{"role": "user", "content": PROMPT + diff}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        f"{BASE_URL}/chat/completions",
        data=body,
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {API_KEY}"},
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        payload = json.load(resp)
    text = payload["choices"][0]["message"]["content"].strip()
    # Models sometimes wrap JSON in fences despite instructions.
    if text.startswith("```

"):
        text = text.split("\n", 1)[1].rsplit("

```", 1)[0]
    try:
        findings = json.loads(text)
    except json.JSONDecodeError:
        # Unparseable output = inconclusive, not a pass.
        print("AI review returned unparseable output; treating as inconclusive.")
        sys.exit(2)
    if not isinstance(findings, list):
        sys.exit(2)
    return findings

def main() -> None:
    base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
    diff = get_diff(base)
    if not diff.strip():
        sys.exit(0)  # nothing to review
    findings = call_model(diff)
    blocking = [f for f in findings if f.get("severity") in BLOCK_AT]
    for f in findings:
        loc = f"{f.get('file', '?')}:{f.get('line') or '?'}"
        print(f"[{f.get('severity', '?'):>8}] {loc} ({f.get('kind', '?')})")
        print(f"           {f.get('summary', '')}")
        if f.get("suggestion"):
            print(f"           -> {f['suggestion']}")
    if blocking:
        print(f"\nGate FAILED: {len(blocking)} high/critical finding(s).")
        sys.exit(1)
    print(f"\nGate passed ({len(findings)} non-blocking finding(s)).")

if __name__ == "__main__":
    main()

Three design decisions worth calling out, because they're the difference between a gate and a toy:

1. Unparseable output exits 2, not 0. If the model returns prose instead of JSON, the gate is inconclusive, and my hook treats exit 2 as "warn me but let me decide." Exiting 0 there would train me to trust a gate that sometimes isn't running at all.

2. Truncation is loud. Diffs over ~24k characters get truncated with a visible marker, because silent truncation means the model reviewed a different diff than the one I'm pushing. For genuinely huge diffs, the honest answer is to split the push — which the gate nagging you about is arguably a feature.

3. Severity threshold, not zero-findings. Low and medium findings print but don't block. If every style nit failed the gate, I'd disable it within a week. The gate exists to catch the "wait, that's a mutable default argument" class of mistake.

Wiring it into pre-push

# .git/hooks/pre-push (or via pre-commit framework)
#!/bin/sh
python ai_review_gate.py origin/main
code=$?
if [ $code -eq 1 ]; then
  echo "AI review gate failed. Fix findings or push with --no-verify (and own it)."
  exit 1
fi
# exit 2 (inconclusive) warns but allows the push
exit 0

Note the --no-verify escape hatch stays available. A gate with no override is a gate people uninstall; the point is to make skipping it a visible, deliberate act, not to make it impossible.

What I feed it and what I don't

Scenario	Verdict
Solo side projects, every push	Runs unconditionally; cost is zero, false positives are cheap
Work repo with human review anyway	Still useful — catches things before a colleague has to
Diff touching auth/crypto/payments paths	Gate runs, but I treat a "pass" as meaningless; human review required regardless
Diff > ~500 lines changed	Split the change; both the model and your reviewer will miss things
Generated code (migrations, lockfiles, protobuf)	Excluded via the `*.py` pathspec and a per-repo ignore list

Limitations, honestly

It has no project context. The model sees a diff, not your architecture. It will flag things that are fine given your conventions, and miss things that are wrong given your invariants. It catches local mistakes, not design errors.
Severity calibration drifts between models. I tuned BLOCK_AT against the free model I was using; if you switch models, re-check what it calls "high" for a week before trusting the threshold.
Free tiers change. Free access and free servers are availability claims as of when I set this up, not a contract. The script points at an OpenAI-compatible endpoint precisely so that if the free option disappears or its limits stop fitting my usage, I swap AI_REVIEW_BASE_URL and keep the gate.
Non-determinism is real even at temperature 0 across provider-side changes. The gate is a heuristic net, not a proof.

Who should skip this

If your team already has fast, thorough human review with short queues, this adds little — the human pass subsumes it. If your diffs are routinely enormous, fix that first; no review method, human or machine, survives a 2,000-line diff. And if you're in a regulated environment where code can't leave your perimeter, a hosted free endpoint is the wrong tool entirely — run a local model or don't do this.

Where this leaves the gate

My pre-push stack is now: lint and type checks (fast, deterministic), then this AI pass (slower, probabilistic), then a human for anything that matters. The AI layer catches a different class of mistake than the linters — "this logic inverts the flag" rather than "this line is too long" — and at zero marginal cost per push, the trade is easy.

If you try it, the thing I'd most like to hear is where the severity threshold lands for you. My guess is "high" is right for solo work and wrong for teams, but I only have my own repos to judge from. If you want to run the same setup, MonkeyCode's free tier is what I pointed the script at; the README for whatever endpoint you choose will have the base URL and model identifier to drop into the env vars.

A Pre-Push Gate for Python Diffs That Fails Loud, Not Often

Finley Sun — Wed, 05 Aug 2026 07:04:34 +0000

The annoying part of AI-assisted review is not getting an answer. It is getting ten vague comments after CI already passed, or a confident red flag with no file, no line, and no way to reproduce it. For small Python services I wanted the opposite: a local gate that runs before git push, keeps deterministic checks first, asks an LLM only about the current diff, and fails only when the finding is specific enough to act on.

The artifact below is a single Python pre-push hook. It does three things in order: run fast local checks, send only the push diff to an OpenAI-compatible endpoint, and write a provenance record under .git/llm-gate/ so a teammate can see what was sent and why the hook blocked. The provider is hidden behind environment variables, so the same hook works with a local server or a hosted endpoint. For a no-cost try while keeping the provider swappable, I used MonkeyCode's free model access and free server option as the endpoint behind those variables; treat availability as something to confirm in your own account before relying on it. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What the hook optimizes for

Deterministic checks win. Formatting, lint, and a tiny test slice are cheaper and more stable than any model.
The model sees the diff, not the repo. That keeps cost, prompt size, and leakage surface down.
Blocking requires structure. A comment must include severity, path, line, rationale, and a suggested patch or check.
Every decision leaves a local artifact. If it blocks at 2 a.m., you can inspect exactly what happened.

Install

Save this as .git/hooks/pre-push and chmod +x it. Do not put keys in the script. Use LLM_BASE_URL, LLM_API_KEY if your endpoint needs one, LLM_MODEL, and optional LLM_GATE_STRICT=1.

#!/usr/bin/env python3
import json
import os
import subprocess
import sys
import urllib.request
from pathlib import Path

ROOT = Path(subprocess.run(['git', 'rev-parse', '--show-toplevel'], capture_output=True, text=True, check=True).stdout.strip())
LOGDIR = ROOT / '.git' / 'llm-gate'
LOGDIR.mkdir(parents=True, exist_ok=True)

def sh(cmd, check=False):
    return subprocess.run(cmd, cwd=ROOT, capture_output=True, text=True, check=check)

def run_deterministic():
    findings = []
    if (ROOT / 'pyproject.toml').exists() and sh(['python', '-m', 'ruff', '--version']).returncode == 0:
        r = sh(['python', '-m', 'ruff', 'check', '.'])
        if r.returncode != 0:
            findings.append({'tool': 'ruff', 'detail': r.stdout + r.stderr})
    if os.environ.get('GATE_FAST_TESTS') == '1':
        r = sh(['python', '-m', 'pytest', '-q', '-x'])
        if r.returncode != 0:
            findings.append({'tool': 'pytest', 'detail': r.stdout[-4000:] + r.stderr[-1000:]})
    return findings

def push_diff(stdin_text):
    # pre-push receives: local_ref local_sha remote_ref remote_sha per line
    diffs = []
    for line in stdin_text.splitlines():
        parts = line.split()
        if len(parts) != 4:
            continue
        local_ref, local_sha, remote_ref, remote_sha = parts
        if local_sha.startswith('0000000'):
            continue
        if remote_sha.startswith('0000000'):
            base = sh(['git', 'merge-base', local_sha, remote_ref]).stdout.strip() or local_sha + '^'
        else:
            base = remote_sha
        d = sh(['git', 'diff', '--unified=20', '--no-ext-diff', base, local_sha, '--', '*.py'])
        if d.stdout:
            diffs.append(d.stdout)
    return '\n'.join(diffs)[:60000]

def ask_model(diff):
    base = os.environ.get('LLM_BASE_URL', '').rstrip('/')
    model = os.environ.get('LLM_MODEL', '')
    if not base or not model or not diff.strip():
        return {'skipped': True, 'reason': 'missing endpoint, model, or diff'}
    prompt = '''Review this Python git diff. Return only JSON with key findings.
Each finding must be: severity high|medium|low, path, line, rationale, suggested_check_or_patch.
Block-worthy means high severity only: correctness bug, data loss, authz/authn bypass, secret leak, unsafe deserialization, or migration risk.
If none, return findings as an empty list. Diff:\n''' + diff
    payload = {'model': model, 'messages': [{'role': 'user', 'content': prompt}], 'temperature': 0}
    req = urllib.request.Request(base + '/v1/chat/completions', data=json.dumps(payload).encode(), headers={'Content-Type': 'application/json'})
    key = os.environ.get('LLM_API_KEY')
    if key:
        req.add_header('Authorization', 'Bearer ' + key)
    with urllib.request.urlopen(req, timeout=45) as resp:
        data = json.loads(resp.read().decode())
    text = data['choices'][0]['message']['content']
    try:
        return json.loads(text)
    except Exception:
        return {'parse_error': True, 'raw': text[:4000]}

def main():
    stdin_text = sys.stdin.read()
    det = run_deterministic()
    diff = push_diff(stdin_text)
    model_out = ask_model(diff)
    record = {'deterministic': det, 'model': model_out, 'diff_chars': len(diff)}
    (LOGDIR / 'last.json').write_text(json.dumps(record, indent=2))

    if det:
        print('pre-push blocked by deterministic checks; see .git/llm-gate/last.json', file=sys.stderr)
        sys.exit(1)
    highs = [f for f in model_out.get('findings', []) if f.get('severity') == 'high']
    if highs and os.environ.get('LLM_GATE_STRICT', '1') == '1':
        for f in highs:
            print('HIGH:', f.get('path'), f.get('line'), f.get('rationale'), file=sys.stderr)
        print('Fix, override with LLM_GATE_STRICT=0 for this push only, or inspect .git/llm-gate/last.json', file=sys.stderr)
        sys.exit(1)

if __name__ == '__main__':
    main()

Reproducible test plan

Create a branch with a deliberate bug: change timeout = 30 to timeout = '30' in a function that passes it to a socket call.
Run git push to a scratch remote. Expect ruff or tests to catch type-adjacent mistakes first when enabled.
Comment out deterministic failures and retry. The model lane should return JSON; if it returns prose, the hook records parse_error instead of guessing.
Add a fake high finding in last.json? Do not. Instead temporarily set LLM_GATE_STRICT=0 and confirm the push proceeds while the artifact remains.
Restore strict mode and verify a clean diff produces findings: [] and no network call when there is no Python diff.

Decision table

Situation	Use this hook?	Why
Solo Python repo, pushes daily	Yes	Fast feedback before CI and review
Monorepo with huge generated diffs	Only with path filters	Diff truncation can hide context
Regulated code with export limits	Maybe not hosted lane	Keep deterministic lane; disable network lane
Team with no prompt/log policy	No	Agree on data retention and override rules first
You need final security signoff	No	This is a gate, not an audit

Limitations

An LLM can miss a real bug or invent a plausible one, so the hook treats it as an advisory layer unless a finding is structured and high severity. Diff-only review lacks architecture context; a safe-looking change can still break an invariant elsewhere. Free tiers and free servers can change, rate limits can appear, and endpoints can be unavailable, so the script must degrade to deterministic checks rather than become a new flaky dependency. Do not send proprietary code to any endpoint until retention, region, and access controls are acceptable.

If you try it, start non-strict for a week, tune the rubric from the false positives you actually see, and only then let high-severity structured findings block pushes.