DEV Community

Cover image for Stop Generating Changelogs from Git Commits: How I Fixed the 'WHY' Problem in ChangelogBot
𝗝𝗼𝗡𝗻
𝗝𝗼𝗡𝗻

Posted on AI-assisted

Stop Generating Changelogs from Git Commits: How I Fixed the 'WHY' Problem in ChangelogBot

#github #typescript #devops #opensource

I recently audited a release page for a popular open-source project and found a 140-line changelog that read like this:

* fix: typo in auth handler (#412)
* refactor: rewrite query builder (#415)
* feat: increase internal timeout to 15s (#418)
Enter fullscreen mode Exit fullscreen mode

As a developer upgrading that package, those lines tell me almost nothing. Why was the query builder rewritten? Did it change memory consumption or SQL dialect support? Why was the timeout tripled? Was there an underlying socket leak under load?

Generating release notes directly from raw git commits or PR titles is one of the most widespread anti-patterns in modern DevOps. We treat commit logsβ€”which are micro-level implementation breadcrumbs meant for authorsβ€”as user-facing documentation.

In this article, I will explain why traditional automated changelogs fail, introduce ChangelogBotβ€”an open-source tool built to extract the actual intent ("WHY") behind changes using AIβ€”and walk through how I re-engineered its GitHub metadata engine to survive secondary rate limits and silent CI failures.

Terminal displaying clean source code on a modern developer workstation

The Mistake: Git Commits Are Not Release Notes

Conventional Commits (feat:, fix:, chore:) solved automated semantic versioning, but they created a massive documentation blind spot.

When developers open pull requests, the richest context lives in the PR description, linked issues, and discussion threads. That is where engineers explain edge cases, breaking behavioral nuances, and performance trade-offs.

However, standard release generators throw all of that away:

  1. They prioritize the what over the why: A commit title like fix: cache invalidation on tenant switch describes code motion, not the customer-facing impact.
  2. They flood readers with internal noise: Squash merges, dependency bumps, and formatting fixes bury critical behavioral changes.
  3. They force consumers to hunt through PR history: If a changelog does not summarize the rationale, developers must click dozens of GitHub links just to evaluate an upgrade.

A real changelog should communicate intent and impact.

Enter ChangelogBot: Extracting the "WHY" with AI

To solve this problem, ChangelogBot takes a radically different architectural approach:

ChangelogBot Banner

Instead of dumping commit messages into Markdown, ChangelogBot:

  • Maps release commits to their original GitHub Pull Requests.
  • Fetches the PR body, author, and associated discussions.
  • Runs a local heuristic trust score to filter out trivial or low-quality descriptions.
  • Leverages LLMs (Anthropic Claude or OpenAI) to extract and summarize the architectural "WHY" behind the change.
  • Categorizes changes cleanly into human-readable sections, linking back to authors and pull requests.

Here is what the difference looks like in practice:

Raw Commit Output (Standard Tools) ChangelogBot Output
* fix: listen to published event (#12) Releases: Listen to the published release event instead of created (#12) by @alice
Why: Draft releases can be created first and published later; listening only to creation missed scheduled production deployments.
* refactor: mapCommitsToPrs (#45) GitHub API: Batch commit-to-PR resolutions through a bounded worker pool (#45) by @johnnylemonny
Why: Firing unbounded parallel requests triggered GitHub abuse detection in repositories with over 30 PRs.

The Production Trap: Where Release Automation Fails Under Load

When I began contributing to ChangelogBot, the core AI extraction worked brilliantly on small repositories. But when I ran it against real production releases containing 40+ merged PRs, the tool hit two critical failure modes in CI/CD.

1. The Unbounded Promise.all Trap (Secondary Rate Limits)

The original commit resolver fetched pull request metadata for every commit in parallel:

// ANTI-PATTERN: Firing unbounded requests simultaneously
const prResults = await Promise.all(
  commits.map(commit => fetchPRDetails(commit.sha))
);
Enter fullscreen mode Exit fullscreen mode

If a release contained 60 commits, this line fired 60 HTTPS requests to api.github.com in the exact same millisecond.

GitHub has a generous primary rate limit (5,000 requests/hour for authenticated users). However, GitHub also enforces Secondary Rate Limits (abuse detection mechanisms designed to prevent concurrent traffic spikes).

The result? The job failed with HTTP 403 Forbidden or HTTP 429 Too Many Requestsβ€”even though the token had over 4,800 hourly requests remaining!

2. Silent Degradation in CI

The second problem was defensive error handling gone wrong. To prevent crashes, previous code wrapped API lookups in broad try/catch blocks and returned empty arrays on failure:

// DANGEROUS: Silent fallback hiding network & auth errors
try {
  return await fetchPRDetails(commitSha);
} catch {
  return []; // CI exits 0, but changelog loses all PR context!
}
Enter fullscreen mode Exit fullscreen mode

When GitHub throttled the pipeline or an API token lacked permissions, the bot did not fail. Instead, it silently dropped the PR associations, skipped the LLM enrichment, and generated a degraded changelog containing only bare commit SHAs. The GitHub Action finished with a green checkmark (exit 0), and the team only noticed the ruined changelog after shipping the release.

How I Re-Engineered the Metadata Pipeline

To make ChangelogBot rock-solid for large enterprise repositories, I designed and implemented a three-part architectural fix.

Step 1: Lightweight Bounded Concurrency (mapConcurrent)

I avoided pulling heavy external dependencies like p-limit and instead implemented a native, lightweight bounded concurrency worker directly in TypeScript:

/**
 * Maps items through an asynchronous mapper with a strictly bounded pool of in-flight executions.
 */
export async function mapConcurrent<Item, Result>(
  items: readonly Item[],
  limit: number,
  mapper: (item: Item, index: number) => Promise<Result>
): Promise<Result[]> {
  const boundedLimit = Math.max(1, Math.min(limit, items.length || 1));
  const results = new Array<Result>(items.length);
  let nextIndex = 0;

  const workers = Array.from({ length: boundedLimit }, async () => {
    while (nextIndex < items.length) {
      const currentIndex = nextIndex++;
      results[currentIndex] = await mapper(items[currentIndex]!, currentIndex);
    }
  });

  await Promise.all(workers);
  return results;
}
Enter fullscreen mode Exit fullscreen mode

By capping the concurrency at GITHUB_METADATA_CONCURRENCY = 4, API lookups remain blisteringly fast while completely eliminating secondary rate-limit triggers.

Step 2: Discriminated Metadata Results

Instead of returning null or empty arrays on failure, I introduced typed result discrimination. Every API interaction now explicitly declares whether it succeeded or degraded, along with a structured failure reason:

export type MetadataLookupFailureReason =
  | 'not-found'
  | 'rate-limited'
  | 'unauthenticated'
  | 'invalid-response'
  | 'network-error';

export type MetadataLookupResult<T> =
  | { status: 'success'; data: T }
  | { status: 'degraded'; reason: MetadataLookupFailureReason; error?: unknown };
Enter fullscreen mode Exit fullscreen mode

Now, the engine cleanly distinguishes between expected missing data (like a 404 for a commit that was rebased or cherry-picked) and critical operational failures (such as rate limits or invalid tokens).

Step 3: Transparent Diagnostic Reporting

I surfaced these metrics directly into the --dry-run console output and the --dry-run-json-report schema:

Metadata lookups: total=24, successful=23, degraded=1
Metadata fallback reasons: Commit 3a1b2c4: PR lookup returned 404 (not found)
Enter fullscreen mode Exit fullscreen mode

In automated CI pipelines, teams can inspect the JSON payload and assert that degradedLookups === 0 before publishing a release to production.

Quick Start: Using ChangelogBot in Your Project

You can run ChangelogBot locally or wire it directly into your GitHub Actions release workflow.

Local Dry-Run 

# Clone and build, or run via npx
npx @necoz-oss/changelog-bot \
  --release-tag v1.4.0 \
  --release-name 1.4.0 \
  --provider anthropic \
  --dry-run
Enter fullscreen mode Exit fullscreen mode

GitHub Actions Workflow

Here is a minimal workflow that automatically updates CHANGELOG.md and creates a pull request when a release tag is pushed:

name: Update Changelog

on:
  push:
    tags:
      - 'v*'

jobs:
  changelog:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Generate Changelog with ChangelogBot
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          REPO_FULL_NAME: ${{ github.repository }}
        run: |
          npx @necoz-oss/changelog-bot \
            --release-tag ${{ github.ref_name }} \
            --release-name ${{ github.ref_name }} \
            --provider anthropic
Enter fullscreen mode Exit fullscreen mode

The Takeaway

Your changelog is often the only piece of documentation your users and downstream developers read during an upgrade.

  1. Stop using commit headers as release notes. Raw commits explain file modifications; your users need to understand technical and product intent.
  2. Never fire unbounded Promise.all against third-party APIs in CI. Secondary rate limits will penalize concurrent bursts even when your hourly quota is full.
  3. Say no to silent try/catch fallbacks. Discriminated union types and structured diagnostics ensure that your automation alerts you when upstream metadata degrades.

Check out the project on GitHub: necoz-oss/changelog-bot. Give it a star, test it on your next release, and see what automated release notes look like when they actually explain the "WHY".

What failure mode has broken an automated release in your CI/CD pipelineβ€”and how did you catch it?

Connect with Me

If you found this article helpful, let's connect!

johnnylemonny GitHub Banner

johnnylemonny image

𝗝𝗼𝗡𝗻

Freelance fullstack developer creating fast, modern, and refined applications. I focus on simplicity, performance, and real value for users. I also write tech-related articles here.

Top comments (2)

Collapse
 
nyaomaru profile image
nyaomaru •

This is such a great write-up! 😸

I really like how you explained both sides of ChangelogBot.

Why release notes should explain the WHY, not just repeat commit messages, and what it actually takes to make that work reliably in real CI environments.

The bounded concurrency and degraded metadata handling were important improvements for the project, so thank you again for contributing them and for documenting the reasoning behind them so clearly! πŸ™Œ

I’m really happy to see ChangelogBot evolving through contributions like this. 🐈

Collapse
 
johnnylemonny profile image
𝗝𝗼𝗡𝗻 •

Thanks so much, @nyaomaru! It was a real pleasure collaborating on this and helping make the metadata pipeline rock-solid for bigger releases. I really appreciate your thorough reviews across the PRs and look forward to seeing the project continue to grow.