I really like how you explained both sides of ChangelogBot.
Why release notes should explain the WHY, not just repeat commit messages, and what it actually takes to make that work reliably in real CI environments.
The bounded concurrency and degraded metadata handling were important improvements for the project, so thank you again for contributing them and for documenting the reasoning behind them so clearly! π
Iβm really happy to see ChangelogBot evolving through contributions like this. π
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Thanks so much, @nyaomaru! It was a real pleasure collaborating on this and helping make the metadata pipeline rock-solid for bigger releases. I really appreciate your thorough reviews across the PRs and look forward to seeing the project continue to grow.
DemoVexa is a desktop app that reads your code and turns it into finished product demo videos, with a script, voiceover and your real UI animated. No screen recording.
The gap between a Conventional Commit title and what someone upgrading needs is the part that lands hardest. A line like "refactor: rewrite query builder" helps the author and tells the upgrader almost nothing about memory use or dialect support. Pulling the WHY from the PR body and linked issues is the right move. I'd add a short "who this affects" note next to each user facing entry, even one sentence. That note is also the seed for a release video later, because the narration has to answer the same question. Surfacing silent CI that writes a half finished file as an error is a real fix too.
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Adding a "who this affects" note is a brilliant next step for the output schema. Even when an entry explains the architectural "why", an upgrader scanning 30 lines just wants to know immediately if their specific setup or SQL dialect is in the blast radius, and using that same summary as the script backbone for release announcements is a great bonus. Catching those silent exit 0 fallbacks in CI was honestly just as important as the prompt itself, since a great extraction prompt doesn't help much when rate limits quietly strip out the PR context beforehand.
DemoVexa is a desktop app that reads your code and turns it into finished product demo videos, with a script, voiceover and your real UI animated. No screen recording.
Glad the "who this affects" note landed. Scanning for blast radius is exactly what an upgrader does in the first pass, and that same sentence is a clean opening line for a release clip later. The silent exit 0 case is the nasty one: the job looks green while the WHY context never made it into the file. When you add that note to the schema, do you plan to make it required for every user-facing entry, or only when the PR body already names a dialect, setup, or migration risk?
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Making it strictly conditional is definitely the safer play here. If the schema forces a required field on every entry, the LLM will inevitably start inventing generic filler like "Affects all users of the library" whenever a PR body doesn't mention a specific constraint, which just adds noise back into the scan. Treating it as an optional field that only populates when the PR text or diff explicitly names a specific runtime, configuration flag, or migration path keeps the signal high and prevents hallucinated scope.
DemoVexa is a desktop app that reads your code and turns it into finished product demo videos, with a script, voiceover and your real UI animated. No screen recording.
Yes, optional is the right call. A forced field just trades one kind of noise for another, and "Affects all users" is worse than a blank because it looks authoritative. Populating only when the PR or diff names a concrete constraint keeps the blast-radius line trustworthy. When you detect that signal, do you look only at the PR body, or also at labels and the diff path list?
Creator of GithubStarsManager β an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
The part that stuck with me is the dry-run report as a gate, not just a log. Asserting degradedLookups === 0 before publishing turns a silent failure into a blocked release, which is exactly the difference between knowing something went wrong and catching it in time. I also liked that a missing WHY stays visibly missing instead of being filled in with a confident guess β that honesty is what makes the rest of the changelog trustworthy.
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Glad those two pieces stood out to you. Turning the dry-run JSON into a hard CI gate and refusing to guess missing context were both about making sure the output is actually reliable in production. Once an AI tool sneaks in a single plausible-sounding hallucination, developers quickly lose trust in the entire changelog.
Creator of GithubStarsManager β an open-source, local-first tool that organizes GitHub stars with AI: auto-categorization, semantic search, release tracking, and an MCP server for star library.
Exactly β that trust point is the one that stuck with me most. A changelog that's right 95% of the time but confidently wrong the other 5% is worse than a sparse one, because you stop checking it at all. The gate plus the visibly-missing WHY feels like the right trade: it fails loudly instead of publishing something smooth-sounding that nobody can trust. Curious β in practice, do you find most of those degraded lookups come from genuinely missing context, or from the model not looking hard enough in the first place?
This is especially true for desktop end-users, who are not reading your commit history: an Electron app's release note that says 'refactor IPC layer' tells a user nothing about whether their crash is fixed or whether the update is safe to install. The shift from implementation breadcrumbs to user-impact notes is the same one we made for auto-update changelogs β grouping changes under fixed, changed, and known issues with one-line plain-language summaries. The rate-limit hardening detail is also a real gotcha: a release-note generator that silently skips a failed fetch produces a thinner changelog with no signal that anything went wrong.
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Desktop and Electron apps are a prime example of where commit-driven changelogs fall flat. End-users deciding whether to click "Update" only care if a release fixes their issue or changes their workflow, not how the IPC layer was refactored under the hood. And catching those silent rate-limit drops was huge for us. A partial changelog that looks complete is one of the easiest ways to accidentally hide important release notes.
We went the other way round at work: the change log is written before the code. Each release gets its change log, product update and press release at the start of development, and they go out to everyone client-facing, the devs and QA together. The why is decided first, so nobody has to dig it out of PR bodies later. Your trust score covers the cases where nobody wrote it down. What does ChangelogBot write when a PR body fails it: a guess, or a visible gap?
Writing the WHY before implementation is probably the nice solution when the whole team can work that way.
One small clarification about ChangelogBot though: the trust score doesnβt try to fill in a WHY that nobody wrote down. Itβs actually there to prevent that kind of guessing.
If a PR body is empty, has no usable WHY candidate, or has a low local trust score, it gets skipped before reaching the model. And if the modelβs result doesnβt meet the required confidence, that WHY is skipped too.
So the changelog entry remains, but ChangelogBot wonβt invent a rationale for it. In dry-run diagnostics, the skip/fallback reason is visible so you can see why enrichment was omitted.
Iβd much rather have a visible missing WHY than a confident-sounding hallucination. πΉ
A visible skip beats a confident guess, agreed. Our agent pipeline leans the same way: every tool call its guard refuses comes back with the reason, so the agent corrects course instead of guessing around a silent no. Does the skip reason only show in dry-run, or does a normal run surface it too, say in the CI summary?
Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.
Right now, per-PR skip reasons (fallbackReasons) only show up in --dry-run and --dry-run-json-report. A normal run stays quiet so it doesn't clutter the logs, and it only drops a "Generated without LLM. Reason: ..." note into the changelog PR body if the entire LLM pass falls back. For CI, the usual setup is running --dry-run-json-report as a quick check before the actual release step.
We're a place where coders share, stay up-to-date and grow their careers.
This is such a great write-up! πΈ
I really like how you explained both sides of ChangelogBot.
Why release notes should explain the WHY, not just repeat commit messages, and what it actually takes to make that work reliably in real CI environments.
The bounded concurrency and degraded metadata handling were important improvements for the project, so thank you again for contributing them and for documenting the reasoning behind them so clearly! π
Iβm really happy to see ChangelogBot evolving through contributions like this. π
Thanks so much, @nyaomaru! It was a real pleasure collaborating on this and helping make the metadata pipeline rock-solid for bigger releases. I really appreciate your thorough reviews across the PRs and look forward to seeing the project continue to grow.
Iβm really happy I got to build this together with you too, John! πΈπ
Letβs keep building and improving it! ππ
The gap between a Conventional Commit title and what someone upgrading needs is the part that lands hardest. A line like "refactor: rewrite query builder" helps the author and tells the upgrader almost nothing about memory use or dialect support. Pulling the WHY from the PR body and linked issues is the right move. I'd add a short "who this affects" note next to each user facing entry, even one sentence. That note is also the seed for a release video later, because the narration has to answer the same question. Surfacing silent CI that writes a half finished file as an error is a real fix too.
Adding a "who this affects" note is a brilliant next step for the output schema. Even when an entry explains the architectural "why", an upgrader scanning 30 lines just wants to know immediately if their specific setup or SQL dialect is in the blast radius, and using that same summary as the script backbone for release announcements is a great bonus. Catching those silent
exit 0fallbacks in CI was honestly just as important as the prompt itself, since a great extraction prompt doesn't help much when rate limits quietly strip out the PR context beforehand.Glad the "who this affects" note landed. Scanning for blast radius is exactly what an upgrader does in the first pass, and that same sentence is a clean opening line for a release clip later. The silent exit 0 case is the nasty one: the job looks green while the WHY context never made it into the file. When you add that note to the schema, do you plan to make it required for every user-facing entry, or only when the PR body already names a dialect, setup, or migration risk?
Making it strictly conditional is definitely the safer play here. If the schema forces a required field on every entry, the LLM will inevitably start inventing generic filler like "Affects all users of the library" whenever a PR body doesn't mention a specific constraint, which just adds noise back into the scan. Treating it as an optional field that only populates when the PR text or diff explicitly names a specific runtime, configuration flag, or migration path keeps the signal high and prevents hallucinated scope.
Yes, optional is the right call. A forced field just trades one kind of noise for another, and "Affects all users" is worse than a blank because it looks authoritative. Populating only when the PR or diff names a concrete constraint keeps the blast-radius line trustworthy. When you detect that signal, do you look only at the PR body, or also at labels and the diff path list?
The part that stuck with me is the dry-run report as a gate, not just a log. Asserting degradedLookups === 0 before publishing turns a silent failure into a blocked release, which is exactly the difference between knowing something went wrong and catching it in time. I also liked that a missing WHY stays visibly missing instead of being filled in with a confident guess β that honesty is what makes the rest of the changelog trustworthy.
Glad those two pieces stood out to you. Turning the dry-run JSON into a hard CI gate and refusing to guess missing context were both about making sure the output is actually reliable in production. Once an AI tool sneaks in a single plausible-sounding hallucination, developers quickly lose trust in the entire changelog.
Exactly β that trust point is the one that stuck with me most. A changelog that's right 95% of the time but confidently wrong the other 5% is worse than a sparse one, because you stop checking it at all. The gate plus the visibly-missing WHY feels like the right trade: it fails loudly instead of publishing something smooth-sounding that nobody can trust. Curious β in practice, do you find most of those degraded lookups come from genuinely missing context, or from the model not looking hard enough in the first place?
This is especially true for desktop end-users, who are not reading your commit history: an Electron app's release note that says 'refactor IPC layer' tells a user nothing about whether their crash is fixed or whether the update is safe to install. The shift from implementation breadcrumbs to user-impact notes is the same one we made for auto-update changelogs β grouping changes under fixed, changed, and known issues with one-line plain-language summaries. The rate-limit hardening detail is also a real gotcha: a release-note generator that silently skips a failed fetch produces a thinner changelog with no signal that anything went wrong.
Desktop and Electron apps are a prime example of where commit-driven changelogs fall flat. End-users deciding whether to click "Update" only care if a release fixes their issue or changes their workflow, not how the IPC layer was refactored under the hood. And catching those silent rate-limit drops was huge for us. A partial changelog that looks complete is one of the easiest ways to accidentally hide important release notes.
We went the other way round at work: the change log is written before the code. Each release gets its change log, product update and press release at the start of development, and they go out to everyone client-facing, the devs and QA together. The why is decided first, so nobody has to dig it out of PR bodies later. Your trust score covers the cases where nobody wrote it down. What does ChangelogBot write when a PR body fails it: a guess, or a visible gap?
I really like that approach! πΈ
Writing the WHY before implementation is probably the nice solution when the whole team can work that way.
One small clarification about ChangelogBot though: the trust score doesnβt try to fill in a WHY that nobody wrote down. Itβs actually there to prevent that kind of guessing.
If a PR body is empty, has no usable WHY candidate, or has a low local trust score, it gets skipped before reaching the model. And if the modelβs result doesnβt meet the required confidence, that WHY is skipped too.
So the changelog entry remains, but
ChangelogBotwonβt invent a rationale for it. In dry-run diagnostics, the skip/fallback reason is visible so you can see why enrichment was omitted.Iβd much rather have a visible missing WHY than a confident-sounding hallucination. πΉ
A visible skip beats a confident guess, agreed. Our agent pipeline leans the same way: every tool call its guard refuses comes back with the reason, so the agent corrects course instead of guessing around a silent no. Does the skip reason only show in dry-run, or does a normal run surface it too, say in the CI summary?
Right now, per-PR skip reasons (
fallbackReasons) only show up in--dry-runand--dry-run-json-report. A normal run stays quiet so it doesn't clutter the logs, and it only drops a"Generated without LLM. Reason: ..."note into the changelog PR body if the entire LLM pass falls back. For CI, the usual setup is running--dry-run-json-reportas a quick check before the actual release step.