Triage OSS Issues Before Cloning: A Deterministic Gate That Cuts Model Calls

Most OSS triage burns model calls on issues a regex can reject in 40 milliseconds. A cheap gate should run first. The model should see only what survives it.

This article describes a pre-clone gate: six text signals, fixed weights, one routing decision. The deterministic stage runs anywhere, including a small remote box. The model stage handles one narrow job.

1. The costly default is clone-first

The default workflow is clone, install, guess, then ask a model. That order spends the expensive step before the cheap one. Half the queue never deserved a worktree.

A triage gate inverts the order. Score the report text first. Clone only what scores high enough to justify it.

2. Six signals, fixed weights

The gate looks for evidence a maintainer would demand anyway. Each signal is a regex over title plus body. Weights are integers, not learned values, so the output is auditable.

#!/usr/bin/env python3
"""triage.py - deterministic pre-clone gate for OSS issues."""
import json
import re
import sys

SIGNALS = {
    "version":    r"\bv?\d+\.\d+(\.\d+)?([-\w.]*)?\b",
    "command":    r"(?m)^\s*(\$|>)\s+\S+",
    "expected":   r"(?i)expected\s*(behaviou?r|result|output)?\s*[:\-]",
    "actual":     r"(?i)(actual|got|instead)\s*(behaviou?r|result|output)?\s*[:\-]",
    "repro_link": r"https?://(gist\.|pastebin|github\.com/\S+/(issues|pull)/|replit|codesandbox)",
    "traceback":  r"(?i)(traceback \(most recent call last\)|error:|exception|panic:)",
}

WEIGHTS = {
    "version": 2,
    "command": 3,
    "expected": 2,
    "actual": 2,
    "repro_link": 4,
    "traceback": 2,
}

SECURITY_LABELS = {"security", "cve", "vulnerability"}


def score(issue: dict) -> tuple[int, dict[str, bool]]:
    text = f"{issue.get('title', '')}\n{issue.get('body', '')}"
    hits = {name: bool(re.search(pat, text)) for name, pat in SIGNALS.items()}
    total = sum(WEIGHTS[name] for name, ok in hits.items() if ok)
    return total, hits


def decide(total: int, issue: dict) -> str:
    labels = {str(l).lower() for l in issue.get("labels", [])}
    if labels & SECURITY_LABELS:
        return "human-first"
    if total >= 9:
        return "reproduce-locally"
    if total >= 5:
        return "ask-two-questions"
    return "model-hypothesis"


def main(path: str) -> None:
    with open(path, encoding="utf-8") as fh:
        issue = json.load(fh)
    total, hits = score(issue)
    out = {
        "issue": issue.get("number"),
        "score": total,
        "max_score": sum(WEIGHTS.values()),
        "signals": hits,
        "route": decide(total, issue),
        "model_calls_allowed": 0 if total >= 9 else 1,
    }
    json.dump(out, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main(sys.argv[1])

The maximum score is 15. The model_calls_allowed field is the point of the whole file. It is a hard budget written next to the evidence.

3. Run the gate and keep the receipt

Fetch the issue as JSON, then run the gate locally. Store the output beside the patch notes.

$ gh issue view 4821 --repo some-org/some-project \
    --json number,title,body,labels > issues/4821.json
$ python3 triage.py issues/4821.json | tee issues/4821.triage.json
{
  "issue": 4821,
  "score": 11,
  "max_score": 15,
  "signals": {
    "version": true,
    "command": true,
    "expected": true,
    "actual": true,
    "repro_link": false,
    "traceback": true
  },
  "route": "reproduce-locally",
  "model_calls_allowed": 0
}

A score of 11 with no repro link still routes to local reproduction. The traceback plus command is enough to try. The model is not invited.

4. Split the work across three places

The gate creates a natural split. Deterministic steps need a shell. Ambiguous steps need a model. Long runs need somewhere other than a laptop.

Stage	Job	Where it runs	Model involved
Gate	Score signals, choose route	Any shell, 40 ms	No
Reproduce	Clone, install, run one command	Local or remote box	No
Verify	Base run plus patched run	Same box, same commit pair	No
Frame	Draft two clarifying questions	Model	Yes, one call
Review	Check findings cite diff lines	Deterministic checker	No

Two of these stages are pure compute. They do not need a model at all. They do need a machine that stays up during a 20-minute install.

This is where MonkeyCode's free model access and free server option fit the workflow described here. Both are operator-supplied availability claims; check the current limits in the product before you plan around them. The useful part for this gate is the split, not the branding: a remote box runs the clone-and-test stages while the model handles only the framing step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

5. Write the handoff contract

The model stage gets a small payload. Sending the whole issue thread invites summary theatre. Send the score, the missing signals, and the failure text only.

{
  "issue": 4821,
  "score": 11,
  "missing_signals": ["repro_link"],
  "route": "reproduce-locally",
  "task": "ask-two-questions",
  "failure_text": "<trimmed traceback, 1200 bytes max>",
  "forbidden": ["summary", "root cause", "patch"]
}

The forbidden list matters. Without it, free-tier models happily produce a confident root cause from a stack trace. That output is worse than nothing, because reviewers read it.

Cap the payload before sending. Strip tokens, paths, and environment dumps first. A secret gate belongs upstream of this step, not inside it.

6. Where the gate breaks

Regex scoring is brittle. Three known failure modes are worth stating plainly.

1. Non-English reports miss most signals and fall to model-hypothesis by default.
2. Issues that describe behaviour in prose without labels score low despite being real bugs.
3. Teams that paste logs into attachments score zero on traceback and command.

The fix is not a smarter regex. The fix is a manual override flag, recorded in the same JSON, so the routing decision stays visible:

$ python3 triage.py issues/4821.json --override reproduce-locally

Security-labelled issues should always route to a human first. No score justifies automating that queue.

7. Who should skip this

Small projects with fewer than ten open issues do not need a gate. Read the queue directly. Projects with a triage rota already have the human equivalent.

Skip this if the team cannot store triage receipts. The value comes from the audit trail, not the score. A number with no recorded evidence is just a vibe with a decimal point.

The gate is roughly 60 lines. It costs one afternoon and saves the expensive step on every low-evidence report. Start with the receipts, then decide whether the model stage earns its place.

If you want to test the remote-box half without provisioning anything, the free server option is the cheapest way to try the split described above.