Building a Free-Tier GitHub Issue Triage Agent: A Case Study

You maintain a small open-source repository. The issues keep coming, and each one needs a label, a category, and a first response. You don't have a budget for a hosted agent, and you don't want to hand your backlog to a closed platform. This case study walks through a small project that solves that: a triage agent that classifies issues, suggests labels, and drafts replies, running entirely on free model access and a free server.

This case study uses MonkeyCode's free model access and free server option to keep the total cost at zero.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Background: the problem

Small repos have a specific pain: the maintainer is also the developer, the reviewer, and the support team. Every new issue interrupts flow. Labels are inconsistent. Replies are repetitive. You need automation, but you also need to trust it.

The constraints are tight. Budget is zero. You don't want to enter a credit card for a trial. You want the agent to run somewhere you control, not on a laptop that sleeps. And you need to know when the agent is wrong before it touches your repo.

Goal: a measurable, small scope

Define a narrow goal before writing code. In this case, the agent should do three things:

1. Classify each issue as bug, feature, or question.
2. Suggest up to two labels from your repo's existing label set.
3. Draft a first-response comment that a human can edit.

Success is measured by three numbers: classification accuracy on a labeled sample, the number of actions that required human confirmation, and the total cost. That's it. No grand promises, no autonomous PRs.

Implementation

1. The agent loop

The core is a minimal function-calling loop. It sends the issue text to the model, reads any tool calls, executes them, and repeats until the model returns a final answer. The script below is an example, not a production library.

# agent.py — example, adapt to your endpoint
import json
import os
import requests

def call_model(messages, tools):
    # Point this at your MonkeyCode endpoint
    resp = requests.post(
        os.environ["MC_ENDPOINT"],
        headers={"Authorization": f"Bearer {os.environ['MC_KEY']}"},
        json={
            "model": os.environ.get("MC_MODEL", "default"),
            "messages": messages,
            "tools": tools,
        },
        timeout=60,
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["message"]

The loop keeps a conversation history and stops when the model stops requesting tools.

def run_triage(issue):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": format_issue(issue)},
    ]
    for _ in range(5):
        msg = call_model(messages, TOOLS)
        messages.append(msg)
        if not msg.get("tool_calls"):
            return msg["content"]
        for tc in msg["tool_calls"]:
            result = execute_tool(tc, dry_run=True)
            messages.append({
                "role": "tool",
                "tool_call_id": tc["id"],
                "content": json.dumps(result),
            })

2. Tools

The tools are deliberately small. Classification and label suggestions are read-only. The reply is drafted but never posted automatically.

TOOLS = [
    {"type": "function", "function": {
        "name": "classify_issue",
        "description": "Classify an issue as bug, feature, or question.",
        "parameters": {"type": "object",
                       "properties": {"category": {"type": "string"}},
                       "required": ["category"]}}},
    {"type": "function", "function": {
        "name": "suggest_labels",
        "description": "Suggest up to two existing labels.",
        "parameters": {"type": "object",
                       "properties": {"labels": {"type": "array",
                                                 "items": {"type": "string"}}},
                       "required": ["labels"]}}},
    {"type": "function", "function": {
        "name": "draft_reply",
        "description": "Draft a first-response comment.",
        "parameters": {"type": "object",
                       "properties": {"body": {"type": "string"}},
                       "required": ["body"]}}},
]

3. The permission gate

This is the part that makes the agent safe to run unattended. Any tool that would change repo state goes through a gate. In dry-run mode, the gate returns needs_confirmation instead of executing.

def execute_tool(tc, dry_run=True):
    name = tc["function"]["name"]
    args = json.loads(tc["function"]["arguments"])
    if name in {"add_label", "comment_on_issue"} and dry_run:
        return {"status": "needs_confirmation", "args": args}
    # real execution only when dry_run=False
    return call_github_api(name, args)

In this case study, the agent never calls a mutating tool directly. The final output is a JSON report that a human reviews.

4. Deploy on a free server

MonkeyCode's free server option is enough for a small scheduled job. The script is stateless: it reads a list of open issues, processes them, and writes a report to a local file. A cron entry runs it hourly.

# example: run every hour, dry-run only
0 * * * * cd /path/to/triage && python agent.py --dry-run >> triage.log 2>&1

Keep the endpoint and model name in environment variables. Free tiers change, and you want to swap models without editing code.

5. Snapshot tests for model swaps

The moment you change models or prompts, behavior drifts. Snapshot tests catch that. Save the tool-call sequence for ten sample issues, then diff against a baseline after any change.

python agent.py --snapshot samples.json --out snapshots/current.json
diff snapshots/baseline.json snapshots/current.json

A diff that changes bug to feature on the same issue is a red flag, even if the new model is cheaper.

Results

What did this case study produce? A repeatable method, not a fixed number. Run the evaluation on your own backlog:

# evaluate.py — example
import json, sys

with open(sys.argv[1]) as f:
    data = json.load(f)  # [{"id": "...", "predicted": "...", "actual": "..."}]

correct = sum(1 for d in data if d["predicted"] == d["actual"])
print(f"accuracy: {correct / len(data):.2%}")

Record three things for every run: accuracy on labeled issues, how many actions hit the permission gate, and the total token cost. In this setup, the cost stayed at zero because both the model access and the server were on free tiers. As of this writing, MonkeyCode's free tier includes 10 million tokens and a free server option; check the project for current details.

Lessons learned

Free models are good enough for structured classification, but only when you constrain the output with tools and a permission gate. The gate is not a nice-to-have. It is the difference between a demo and something you can leave running.

Snapshot tests are cheap insurance. They take ten minutes to set up and they catch regressions before they reach your repository.

Environment variables matter. The endpoint, the key, and the model name should all be configuration, not code. When the free tier changes, you change one line in .env, not your agent logic.

Limitations and who should skip this

This approach is not for everyone. If your issues contain sensitive data, sending them to a third-party model endpoint is a problem. If your repo gets hundreds of issues a day, a free server will not keep up. And if you need fully autonomous label application with no human review, this design is deliberately not for you.

The point of this case study is not to replace human judgment. It is to remove the repetitive part of triage while keeping a human in the loop.

Try it

If you want to test this flow with free model access and a free server, the MonkeyCode project has the current details on its free tier. Start with the dry-run mode and a snapshot test, and you will know quickly whether the approach fits your repo.