Release Notes in a Hurry: A Free-Tier LLM Workflow With an Honest Scoring Script

Every other Tuesday I stare at a blank changelog file.

Monday was a blur of merges. The messages say things like "fix query", "add field", or worse — "update". If I try to write release notes from memory, I miss half the changes. If I dig through every commit, I lose an hour. The worst part is the inconsistency: some notes are one vague sentence, others are three paragraphs about a two-line fix.

So I built a small workflow that drafts release notes from git history and then scores the draft before I read it. It runs entirely on free tiers. Here is the honest engineering log, including the part where the scoring script caught a bad draft that my eyes almost accepted.

The Core Idea

Instead of trusting an AI to produce a final note, I treat it as a first draft generator. A separate script checks the draft against measurable signals. If those signals pass, I review the text. If they fail, I rewrite the prompt or fetch more context.

The pipeline is three steps:

1. Collect commit messages between two tags.
2. Send them to a free model with a strict prompt.
3. Score the output on verb specificity, change names, and repetition.

No database. No frontend. No user accounts. Just a Python script and a tiny API endpoint.

Why MonkeyCode

I needed a place to call an LLM without entering a credit card. MonkeyCode is an open-source project that provides free models through an OpenAI-compatible API, and it also has a free server option for small workloads like this script. For a weekend project with no traffic guarantees, that combination matched my budget: zero.

The setup is refreshingly boring. You point your HTTP client at their base URL, use the same chat completions shape, and pick one of the available models. No SDK is required.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Generator

Here is the script that turns git history into a release note draft.

import subprocess
import requests
import os


def get_commits(start_tag, end_tag):
    """Return commit messages between two tags."""
    result = subprocess.run(
        ["git", "log", "--oneline", f"{start_tag}..{end_tag}"],
        capture_output=True,
        text=True,
        check=True,
    )
    return result.stdout.strip()


def generate_release_notes(commit_text, api_key, base_url, model):
    system_prompt = (
        "You are a technical writer. Convert commit messages into release notes. "
        "Use active verbs. Mention specific endpoints, functions, or files when present. "
        "Group into Added, Fixed, Changed. Do not invent details."
    )

    payload = {
        "model": model,
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": f"Commits:\n{commit_text}"},
        ],
        "temperature": 0.2,
    }

    response = requests.post(
        f"{base_url}/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json=payload,
        timeout=60,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

if __name__ == "__main__":
    start = os.environ["START_TAG"]
    end = os.environ["END_TAG"]
    api_key = os.environ["MONKEYCODE_API_KEY"]
    base_url = os.environ["MONKEYCODE_BASE_URL"]
    model = os.environ.get("MODEL_NAME", "default-free-model")

    commits = get_commits(start, end)
    draft = generate_release_notes(commits, api_key, base_url, model)
    print(draft)

This is intentionally plain. The magic, if you can call it that, lives in two places: the system prompt and the scoring function.

The Scoring Script

A raw LLM draft can look fluent while saying almost nothing. “Improved performance” is a classic. I needed a way to fail those drafts before they reach a human.

The scorer checks three signals:

• Does every sentence have a strong verb (added, fixed, deprecated, removed)?
• Does the draft reference at least one concrete token from the commit text, like a file name or a cache key?
• How repetitive is the phrase structure? A draft where every line starts with “Fixed” is a warning sign.

import re
from collections import Counter

VERBS = ["add", "fix", "change", "remove", "deprecate", "update", "refactor"]

def score_draft(draft, source_commits):
    sentences = [s.strip() for s in re.split(r"[\n.]", draft) if s.strip()]
    if not sentences:
        return 0.0, ["Empty draft"]

    verb_hits = 0
    for s in sentences:
        if any(v in s.lower() for v in VERBS):
            verb_hits += 1

    concrete_tokens = set(re.findall(r"(?:\b[a-zA-Z_]+\.\w+|/\w+/\w+|\b\w+_\w+\b)", source_commits))
    draft_tokens = set(draft.split())
    overlap = len(concrete_tokens & draft_tokens)

    first_words = [s.split()[0].lower() for s in sentences if s.split()]
    repetition_penalty = 0.3 if len(first_words) > 3 else 0.0
    common = Counter(first_words).most_common(1)[0]
    if common[1] / max(len(first_words), 1) > 0.5:
        repetition_penalty = 0.6

    verb_score = verb_hits / len(sentences)
    concrete_score = min(overlap / 2, 1.0)
    final = round(max(0.0, verb_score * 0.5 + concrete_score * 0.5 - repetition_penalty), 2)
    return final, ["Low verb density", "No concrete tokens", "High repetition"]

That function is deliberately heuristic. It will not catch every bad note. But it caught the worst one from my test run: a draft that repeated “Fixed” six times and never mentioned a single component. My eyes had already started skimming it. The scorer gave it 0.17. I rewrote the prompt to demand component names, and the next run scored 0.82.

Running It on the Free Server

I did not want to keep this script terminal-only. A quick FastAPI wrapper made it callable from anywhere, and MonkeyCode's free server was enough to host it for my small test.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import os

app = FastAPI()

class NoteRequest(BaseModel):
    start_tag: str
    end_tag: str

@app.post("/release-notes")
def release_notes(req: NoteRequest):
    api_key = os.environ["MONKEYCODE_API_KEY"]
    base_url = os.environ["MONKEYCODE_BASE_URL"]
    model = os.environ.get("MODEL_NAME", "default-free-model")

    commits = get_commits(req.start_tag, req.end_tag)
    draft = generate_release_notes(commits, api_key, base_url, model)
    score, warnings = score_draft(draft, commits)
    if score < 0.5:
        raise HTTPException(status_code=422, detail={"score": score, "warnings": warnings})
    return {"score": score, "release_notes": draft}

Deployment was the same as any containerized FastAPI app. I set three environment variables and pushed. The first request took several seconds because the server was cold. After that, responses were acceptable for a hobby tool.

A Real-World Test

I tested the workflow on a project with six commits between two releases. The commits contained messages like “refactor session cache into its own module” and “fix offset pagination in /v1/items”. The first draft said:

Fixed issues in the codebase and improved stability.

The scorers verb density was 100%, but the concrete-token overlap was zero. It caught what a human would have carefully missed after a long day. A second attempt with a stronger prompt produced:

Added a dedicated session cache module. Fixed offset pagination in the /v1/items endpoint by validating the limit parameter.

Short. Specific. Correct. That is the level I would write after twenty minutes, and the script did it in forty seconds.

When This Workflow Helps

Situation	Use this?	Why
Solo side project, irregular releases	Yes	Saves drafting time, human still reviews
Team release notes with strict wording	No	Compliance language needs manual control
Rapid hotfix documentation	Maybe	Only if a human verifies the diff
Teaching people to write better notes	Yes	The scoring script makes style issues visible
High-traffic public API changelog	No	Free-tier latency and variance are risks

Limitations and Who Should Not Use This

The free model access comes with unspecified rate limits. My test had a few timeouts, so the script needs retry logic if you run it daily. The free server may go cold after inactivity; the first call after a pause can be slow. Also, this generator will hallucinate details if the commit messages themselves are vague. It cannot invent a missing description.

Do not use this workflow if your release notes are legal documents or need approved vocabulary. Do not use it if every note must be byte-for-byte approved by a change advisory board. For those cases, a human is still the right compiler.

Final Notes

What I learned is not that LLMs write great release notes on their own. They do not. They write plausible drafts, and my scoring script gave me a cheap reason to reject the bad ones. The combo — free models for generation, a free server for hosting, and a ten-line heuristic for control — turned a 40-minute chore into a three-minute review.

If you want to try this loop, MonkeyCode's free tier is a low-risk place to start. I only spent CPU cycles, not card balance. But keep the scorer. Your future self will thank you when the AI confidently says “improved performance” and you know better.