Every release note is a small act of trust, but where does that trust come from when the author writes with total confidence and only the context you just handed it? I spent two days answering that question with MonkeyCode's free model access and a free server option for the scheduled job that runs on every merge. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The plan was simple: turn real diffs into changelog entries, then check every single claim against the diff itself.
The 48-Hour Setup
The pipeline was deliberately boring: a merge triggers a webhook, the free server pulls the new tag, and a script hands the model a compact diff summary. The model had exactly one job: return a JSON object with a release summary, a list of touched files, and a suggested semver bump. I did not ask it to write code, and I did not ask it to review the code; I asked it to describe what changed and nothing more.
That schema turned out to be the only part of the system that survived both days without modification:
{
"summary": "short release summary",
"changed_files": ["path/to/file.py"],
"semver_bump": "major|minor|patch|none",
"confidence": "high|medium|low"
}
Day One — The Filenames That Never Existed
Within the first six hours the model referenced src/analytics.ts in a repository where that file has never existed, and it did so with perfect grammar and zero hesitation. The scary part was not the hallucination itself; the scary part was that the output read like a completely normal release note. A human reviewer would have skimmed it, nodded, and shipped it to customers.
That is when I added the grounding checker, a small script that refuses to trust anything the model writes unless it can trace the claim to the diff. It compares every changed file against git diff --name-only, greps the changed files for identifiers mentioned in the summary, and rejects the entire entry when a single symbol is missing:
def ground_check(model_output, diff_files):
missing_files = set(model_output["changed_files"]) - set(diff_files)
if missing_files:
return False, f"files not in diff: {missing_files}"
for mention in extract_identifiers(model_output["summary"]):
if not grep_changed_files(mention, diff_files):
return False, f"no match for: {mention}"
return True, "grounded"
The checker flagged several ungrounded claims across the two days, and every single one of them looked plausible on first read.
Day Two — The Version Bump That Moved on Its Own
The same diff produced a patch bump at 09:00 and a minor bump at 16:30, and the only thing that changed between those two runs was the batching of the request. Worse, one deleted block containing the comment TODO: remove this entire function was enough to flip the model to major, because it read the word "remove" as a breaking change. I used the semver.org rules as the reference for the bump decision, and the model quietly disagreed with them more than once over those two days.
The fix was to stop treating the model as a decision-maker and start treating it as a witness that proposes. A rule-based function re-derives the bump from known markers, and the changelog keeps the model's wording only when the rule agrees; when they disagree, the entry is routed to a human instead of being published.
What the 48 Hours Actually Told Me
| What broke | What caught it | Would I repeat it? |
|---|---|---|
| Hallucinated filenames | Grounding checker against the diff | Yes |
| Markdown fences wrapped around JSON | Schema guard plus a retry | Yes |
| Same diff, different semver bump | Rule-based re-derivation | Yes |
| Free server restart in the middle of a run | Idempotency key on the job | Yes |
| Truncated output on very large diffs | Scoped per-PR diffs | Yes |
The only failure the stack caught without any human help was truncation, and that is because I scoped the input per merge instead of per release. If you feed a model a three-hundred-file diff, you are not testing its summarization skills; you are testing its tolerance for pain.
What I Would Keep, and What I Would Cut
- Keep the grounding checker, because a changelog claim that cannot be traced to a diff line is noise, and every sentence in a release note should be evidence.
- Keep the idempotency key, because the free server restarted somewhere after midnight and the job would have written the same entry twice without it.
- Keep the human gate on semver disagreements, since the model's confidence score stayed high even when its bump was wrong.
- Cut the idea of using the summary verbatim, because the model's best work was a draft, not a deliverable.
- Cut any ambition to run security-sensitive release notes through the free tier, because a hallucinated file name there costs you more than five minutes.
Who Should Not Copy This
This workflow is for internal changelogs, weekly digests, and demo pipelines where a wrong file name costs you a mild headache. If your release notes go to customers, if they describe security fixes, or if your compliance team reads them, keep the free model away from the final step and keep a named human on the publish button. I would also avoid this approach in a large monorepo, because the grounding check loses its power when a symbol legitimately exists somewhere else in the tree.
Two days later, I still trust the diff more than the summary, and I am convinced that is the correct order of operations. The model made my first draft faster, but the diff was the ground truth the whole time; the changelog only improved once it had a referee. If you try the same experiment, save the raw model output before you parse it, because that raw file is the only evidence you will have after the server restarts.
Top comments (0)