Finally, Release Notes That Don't Suck: Automating with LLMs and GitHub Actions

You know that feeling. It’s Friday afternoon, you’ve just merged a bunch of features and fixes, and now it’s time for the dreaded release notes. You scroll through Git history, try to remember what each commit actually did, then cross-reference Jira to get the full picture. It's tedious, error-prone, and honestly, a soul-crushing task that pulls me away from the fun stuff. For years, I've just… avoided it, or penned the most generic 'bug fixes and improvements' ever.

The Problem: Manual Release Notes Are a Grind

I’m an indie developer. Every minute I spend on administrative overhead is a minute I’m not building. Manual release note generation was a huge time sink. My releases often had a dozen or more tickets, plus countless small commits. Synthesizing all that into something concise, accurate, and actually useful for users felt like pulling teeth. The worst part? Half the time, the notes were boilerplate anyway because I was too lazy (or busy) to dig deep. I knew there had to be a better way.

What I Tried First: Just Feed it to an LLM

My initial thought was pretty simple: large language models are good at summarizing, right? So, I wrote a quick Python script to grab git log --pretty=format:"%s" --since="last_release_tag" and fed the raw output to OpenAI's gpt-3.5-turbo. I figured it would just magically spit out perfect release notes.

It did not. It was a disaster, actually. The output was either:

1. Too generic: "This release includes various updates and bug fixes to improve application stability." (Yeah, thanks, I already said that.)
2. Too verbose: It would often just rephrase every single commit message, leading to paragraphs of developer-speak nobody cared about.
3. Missing context: My commit messages aren't always perfect. "Fix layout" doesn't tell a user which layout or why it was broken. Crucial context from Jira tickets was completely absent.

I tried refining the prompt, telling it to focus on user-facing changes, but without the Jira context, it was essentially guessing. The LLM didn't know if refactor: db migrations was an internal chore or a foundational change enabling a new feature. It needed more.

What Actually Fixed It: Context is King

After a few frustrating hours – probably half a day, just wrestling with the initial approach – I realized the LLM needed more structured, meaningful data. It needed the why behind the what. That meant integrating Jira. Here’s the pipeline I built and wrapped in a GitHub Action:

1. Grab Git Commits and Extract Jira IDs

First, I needed the commit messages since the last release. I use conventional commits (e.g., feat: PROJECT-123 Add user profiles), which made extracting Jira IDs a breeze. I used git log --pretty=format:"%s" "$(git describe --tags --abbrev=0)..HEAD" to get messages between the last tag and HEAD.

Then, a simple regex on the commit messages to find [A-Z]{2,}-\d+ patterns. This gave me a unique list of Jira tickets relevant to the changes.

2. Fetch Jira Details

For each unique Jira ID, I hit the Jira API. I’d previously done this with the jira-python library for local scripts, but for the GitHub Action, a curl call with secrets.JIRA_API_TOKEN felt more straightforward and less dependency-heavy. I fetched the ticket summary, type (Bug, Story, Task), and status. This provided the crucial context.

3. Combine and Prompt the LLM

This was the core. I structured the input for the LLM like this:

text
Here are recent Git commit messages:

• feat: PROJECT-123 Add user profiles
• fix: PROJECT-456 Improve dashboard load time
• chore: internal cleanup

Here are the details for relevant Jira tickets:

• PROJECT-123 (Story): Implement user profile management
• PROJECT-456 (Bug): Dashboard slow on large datasets

Generate concise, user-facing release notes (3-5 bullet points). Focus on new features, bug fixes, and performance improvements. Exclude internal changes. Group similar items.

I found gpt-3.5-turbo-16k was perfectly capable and significantly faster/cheaper than gpt-4 for this task, costing me about $0.05-$0.10 per run. The 16k context window was useful for longer release cycles with many commits.

4. Wrap it in a GitHub Action

I created a Python script that performs steps 1-3, and then wrapped it in a GitHub Action. The action triggers on workflow_dispatch for manual runs or push when a new tag is created. It outputs the generated notes to a file and also as an action output, which can then be used by other steps (e.g., creating a GitHub release draft).

Here's a simplified snippet of the action YAML:

yaml
name: Generate Release Notes
on:
workflow_dispatch:
push:
tags:
- 'v*..'
jobs:
generate_notes:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install dependencies
run: pip install requests # For Jira API
- name: Run Release Note Generator
id: release_notes
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
JIRA_API_TOKEN: ${{ secrets.JIRA_API_TOKEN }}
JIRA_EMAIL: ${{ secrets.JIRA_EMAIL }}
JIRA_BASE_URL: ${{ secrets.JIRA_BASE_URL }}
run: python .github/scripts/generate_release_notes.py
- name: Create Release Draft
uses: softprops/action-gh-release@v1
if: startsWith(github.ref, 'refs/tags/')
with:
tag_name: ${{ github.ref_name }}
body_path: RELEASE_NOTES.md # Output from the script
draft: true

The Payoff: Concise, Accurate, Automated

Now, generating release notes is no longer a chore. I push a tag, the GitHub Action runs, and I get a draft release on GitHub with beautifully summarized notes, cross-referencing both Git and Jira. They're concise, user-focused, and crucially, they reflect what actually changed from a user's perspective. It took some elbow grease to set up, especially getting the prompt just right and handling Jira API authentication securely in the action, but it’s paid off immensely in saved time and mental overhead. My releases are smoother, and my users get better, more informative updates. It feels good to finally tame that beast.