DEV Community

Jordan Huang
Jordan Huang

Posted on

Triage Open-Source Issues With a Model That Only Reads Public Text

#ai #devops #opensource #productivity

When you maintain an open-source project, the issue tracker can turn into a second full-time job. Most reports are useful, but a meaningful slice are duplicates, feature requests disguised as bugs, or reports missing the version and reproduction steps you need to act. You don't want to ignore them, and you don't want to give a bot write access to your repository just to sort them.

A smaller solution is a read-only triage script. It receives the public title and body of a new issue, asks a model to suggest labels and flag possible duplicates, and returns the result as JSON. The script never touches your repository, never closes an issue, and never comments. You remain the only person or process that can take action.

MonkeyCode's free model access can power the classification step, and its free server option can host the script as a short-lived HTTP endpoint for manual or CI-triggered runs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you don't have those options, the same script works with any model endpoint and any Node.js 18+ host, including your laptop.

Start from a problem, not a prompt

Instead of asking the model to "triage this issue," give it a narrow job with a stable output shape. The input is plain text you already have: the issue title and the issue body. The output is three fields:

  • labels: an array of suggested labels chosen from a fixed list.
  • possible_duplicates: an array of short phrases that might match existing issues, or an empty array.
  • needs_more_info: true when the issue lacks a version, a reproduction, or enough detail to act on.

The fixed label list prevents the model from inventing tags. The duplicate field is a search hint, not a link to a specific issue. The needs_more_info flag tells you where to spend your first reply.

This prompt stays the same for every issue, which makes the output easy to verify:

You are triaging a single open-source issue. Do not close, edit, or comment on anything.

Available labels: bug, enhancement, question, documentation, duplicate, needs-repro.

Return ONLY valid JSON with this shape:
{
  "labels": ["..."],
  "possible_duplicates": ["short phrase", "..."],
  "needs_more_info": true
}

Rules:
- Use only labels from the available list.
- possible_duplicates is empty when the issue looks unique.
- needs_more_info is true when version, environment, or reproduction steps are missing.

Issue title: {{TITLE}}

Issue body:
{{BODY}}
Enter fullscreen mode Exit fullscreen mode

A self-contained Node.js script

The script below uses only Node.js built-ins: http for the server and global fetch for the model call. It has no npm dependencies, so it can run on any Node.js 18+ runtime, including a disposable free server.

const http = require('http');

const MODEL_URL = process.env.MODEL_URL || 'http://127.0.0.1:9999/v1/chat/completions';
const PORT = process.env.PORT || 3000;

function buildPrompt(title, body) {
  return `You are triaging a single open-source issue. Do not close, edit, or comment on anything.

Available labels: bug, enhancement, question, documentation, duplicate, needs-repro.

Return ONLY valid JSON with this shape:
{
  "labels": ["..."],
  "possible_duplicates": ["short phrase", "..."],
  "needs_more_info": true
}

Rules:
- Use only labels from the available list.
- possible_duplicates is empty when the issue looks unique.
- needs_more_info is true when version, environment, or reproduction steps are missing.

Issue title: ${title}

Issue body:
${body}`;
}

async function callModel(prompt) {
  // Replace this fetch with your model endpoint. The contract is:
  // request -> { model: string, messages: [{role:'user', content: prompt}] }
  // response -> { choices: [{ message: { content: string } }] }
  const res = await fetch(MODEL_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'local-model',
      messages: [{ role: 'user', content: prompt }],
    }),
  });

  if (!res.ok) {
    throw new Error(`model endpoint returned ${res.status}`);
  }

  const data = await res.json();
  const content = data.choices?.[0]?.message?.content ?? '';
  const jsonStart = content.indexOf('{');
  const jsonEnd = content.lastIndexOf('}');
  if (jsonStart === -1 || jsonEnd === -1) {
    throw new Error('model response did not contain JSON');
  }
  return JSON.parse(content.slice(jsonStart, jsonEnd + 1));
}

const server = http.createServer(async (req, res) => {
  if (req.method !== 'POST' || req.url !== '/triage') {
    res.writeHead(404, { 'Content-Type': 'application/json' });
    res.end(JSON.stringify({ error: 'not found' }));
    return;
  }

  let body = '';
  req.on('data', (chunk) => { body += chunk; });
  req.on('end', async () => {
    try {
      const { title, body: issueBody } = JSON.parse(body);
      if (!title || !issueBody) {
        throw new Error('title and body are required');
      }
      const prompt = buildPrompt(title, issueBody);
      const result = await callModel(prompt);
      res.writeHead(200, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify(result));
    } catch (err) {
      res.writeHead(400, { 'Content-Type': 'application/json' });
      res.end(JSON.stringify({ error: err.message }));
    }
  });
});

server.listen(PORT, () => {
  console.log(`triage endpoint listening on :${PORT}`);
});
Enter fullscreen mode Exit fullscreen mode

Run it locally with MODEL_URL pointed at any OpenAI-compatible endpoint, then test it with curl:

curl -s http://127.0.0.1:3000/triage \
  -H 'Content-Type: application/json' \
  -d '{"title":"App crashes when I click export","body":"It just crashes. No error message."}'
Enter fullscreen mode Exit fullscreen mode

The expected result is needs_more_info: true because the body has no version, environment, or reproduction. The model may also suggest bug and needs-repro, but those are suggestions, not decisions.

Test the triage script before trusting it

A triage helper is only useful if you can predict its behavior. Create four small fixture issues and run them through the script:

Fixture Expected needs_more_info Expected label hint
"Crashes on export" with no details true needs-repro, bug
"Support Python 3.12" with clear context false enhancement
"How do I configure retries?" false question
Duplicate of a known crash report false duplicate, bug

For the duplicate fixture, include a phrase like "same as #142" in the body and check that possible_duplicates contains a short matching phrase. If the model misses it twice in a row, the prompt is too vague or the label list is too broad. Reduce the label list to three options and rerun before deciding the model is unsuitable.

This calibration step matters because a model tuned for general chat will happily return five labels when you asked for two. The fixed JSON shape and the explicit rules in the prompt catch most of those failures before you ever see them in a real issue.

Deployment and limits

The script is a single file, so deploying it to a free server is a matter of exposing PORT and setting MODEL_URL to the model endpoint. If you use a remote host, do not put a repository token, a webhook secret, or any customer data in the environment. The script only needs the public title and body of an issue, and it returns suggestions, not actions.

This approach is not a replacement for a bot that can label, close, or move issues. It will not read a private repository, resolve duplicates against a live issue database, or learn your project's history. It will occasionally confuse a feature request with a question, and it will sometimes miss a duplicate because the duplicate is phrased completely differently. That is why the script returns possible_duplicates as text rather than a link: you search the tracker yourself and confirm the match.

Who should not use this: maintainers of private repositories who need a full access-controlled bot, projects with strict compliance rules around third-party model endpoints, and teams that want automatic issue closure without human review. For those cases, invest in a proper bot with scoped tokens and an approval workflow. For public projects where most issues are readable and the maintainer is the bottleneck, a read-only triage script is a smaller, safer first step than giving a model write access.

Pick one real issue and run it by hand

Before you wire anything into a CI job or a webhook, take five open issues from your tracker that you already triaged by hand. Run them through the script and compare the suggestions to what you actually did. If the script disagrees with you on three of the five, adjust the label list or the prompt until it matches your judgment on the easy cases.

If you have access to MonkeyCode's free model and free server options, start with those five issues on a short-lived endpoint and keep the JSON output in a local file. Once you trust the suggestions, you can decide later whether to connect the script to a read-only GitLab public API call or a manual curl step in your workflow. The important part is that the model never receives write credentials, and you stay in control of every action.

Top comments (0)