DEV Community

Cover image for We Built an AI-Powered Semantic Release Pipeline - Here's Everything We Learned
Smit Vaghasiya
Smit Vaghasiya

Posted on

We Built an AI-Powered Semantic Release Pipeline - Here's Everything We Learned

#github #devops #cicd #ai

We Built an AI-Powered Semantic Release Pipeline (And Survived All the Edge Cases)

If you've ever manually bumped a version number, forgotten to update a changelog, or merged a PR with a hardcoded API key still sitting in the diff — this post is for you.

Over the last few weeks, our team rebuilt our entire release process from scratch. No more manual versioning. No more "did anyone actually review this?" No more emergency hotfixes that skip every safety check. Just a fully automated, AI-assisted pipeline that takes code from a developer's laptop to production — safely, predictably, and with a full audit trail.

Here's exactly how we built it, what broke along the way, and the architecture that finally worked.

The Problem We Started With

Like most growing teams, our release process had accumulated years of manual steps:

  • Developers manually decided what the next version number should be
  • Changelogs were written (or forgotten) after the fact
  • Code review quality varied wildly depending on who reviewed it and how tired they were
  • There was no consistent way to promote code through environments
  • Emergency fixes had no formal approval process — someone just pushed to main and hoped for the best

We wanted a system where:

  1. Version numbers are never manually typed — they're computed automatically from branch names
  2. Every PR gets a consistent, thorough code review — not dependent on who's available
  3. Releases document themselves — no changelog maintenance required
  4. Production changes require real accountability — multiple approvers, no exceptions
  5. Emergencies have a defined process — not "just push it and pray"

The Architecture

We landed on a four-environment pipeline: Development → QA → Staging → Production, each mapped to its own long-lived branch (develop, qa, staging, main).

feature/xyz ──┐
major/xyz  ───┼──► develop ──► qa ──► staging ──► main
hotfix/xyz ───┘                                     ▲
                                          emergency/xyz (bypass)
Enter fullscreen mode Exit fullscreen mode

Branch naming drives everything

We use branch prefixes as the single source of truth for what kind of change is happening:

Prefix Meaning Version Impact
major/ Breaking change X.0.0 (resets Y, Z)
feature/ New functionality x.Y.0 (resets Z)
hotfix/ Bug fix x.y.Z
emergency/ Production incident Separate prod-only counter

A developer never has to think about "what should the version be." They just name their branch correctly, and the pipeline figures out the rest.

AI Code Review That Actually Reads Every File

This was the part we were most skeptical about going in. Could an LLM actually catch real bugs instead of just generating generic "consider adding comments" noise?

Turns out — yes, if you build it right.

What we got wrong at first

Our first version sent the entire PR diff to OpenAI in one blob. Problems:

  • Large PRs got truncated, so files at the end never got reviewed
  • The model would occasionally hallucinate issues that weren't there
  • Everything got mashed into unstructured prose that was hard to act on

What actually worked

We rebuilt it to review file by file, using the GitHub Pull Requests API (/pulls/{number}/files) instead of a raw git diff — this matters more than you'd think, because git diff --name-only between two SHAs can silently miss files when a PR has many commits.

Each file gets its own OpenAI call with a language-aware prompt. A .py file gets checked for bare except: clauses and mutable default arguments. A React component gets checked for dangerouslySetInnerHTML and missing key props. A .env.example file gets checked exclusively for real credentials versus placeholder values.

LANGUAGE_INSTRUCTIONS = {
    "python": "Check for bare except clauses, mutable defaults...",
    "react_jsx": "Check for dangerouslySetInnerHTML, missing key props...",
    "env_file": "Focus entirely on real credentials vs placeholders...",
    # ...12 languages total
}
Enter fullscreen mode Exit fullscreen mode

We also run a regex secret scanner independently of the AI call — 20+ patterns covering AWS keys, Stripe keys, GitHub tokens, JWTs, database connection strings with embedded credentials. This runs first and gets fed into the AI prompt as context, so even if the model misses something, the regex layer catches it.

Temperature is set to 0.05. We wanted consistency and factual accuracy, not creativity. Every finding includes a line number and a suggested code fix in a proper code block:

🔴 **AWS access key** *(line 42)*
Hardcoded AWS access key found — will be exposed in git history.
Enter fullscreen mode Exit fullscreen mode

Suggested fix:
​

# Move to environment variable
aws_key = os.environ["AWS_ACCESS_KEY_ID"]
Enter fullscreen mode Exit fullscreen mode


python

The result gets posted as a single structured PR comment, sorted with critical findings first, using collapsible sections so a 40-file PR review doesn't become an unreadable wall of text.

AI-Generated Semantic Versioning

Version bumping is fully automated using a Python script triggered on every merge to develop:

if pr_branch.startswith("major/"):
    major += 1; minor = 0; patch = 0
elif pr_branch.startswith("feature/"):
    minor += 1; patch = 0
elif pr_branch.startswith("hotfix/"):
    patch += 1
Enter fullscreen mode Exit fullscreen mode

OpenAI then generates the release title and notes based on the PR title, description, and bump type — and everything gets pushed straight into a GitHub Release. We deliberately chose not to maintain a CHANGELOG.md file. GitHub Releases already give you a timestamped, versioned, publicly linkable history — maintaining a parallel markdown file felt redundant.

Production has its own versioning scheme entirely

This is the part I haven't seen many teams do. Production uses a completely separate two-component version — Prod-vA.B — decoupled from the x.y.z development version:

  • A increments when staging is promoted to main (a full release cycle)
  • B increments when an emergency/ branch is merged directly to main (a hotfix outside the normal flow)

So Prod-v3.0 tells you at a glance "this is the third full staging promotion, no emergency patches since." Prod-v3.2 tells you "two emergency fixes have landed since the last full promotion." It's a small thing, but it gives leadership an instant read on production stability without digging through commit history.

The Emergency Release Process

Not every fix can wait for the full develop → qa → staging → main pipeline. Sometimes production is down and you need code live in minutes, not hours.

We built a formal emergency/ branch type that bypasses QA and staging entirely — but it comes with a hard requirement: three-party sign-off. In our case, that's our Business lead, Head of Engineering, and Product Manager, all required to approve on the PR before it can merge to main. No exceptions, no single point of authority.

if [[ "$SOURCE" == emergency/* ]]; then
  echo "branch_type=emergency"
  # Requires 3 approvals before merge is possible
fi
Enter fullscreen mode Exit fullscreen mode

This isn't just process theater — it forces a moment of "are we sure this is actually an emergency" before someone can bypass every normal safety check.

The Hard Lessons (a.k.a. Everything That Broke)

I want to be honest about how much iteration this took, because I think the "we shipped it perfectly on the first try" version of these posts is never true.

Squash merges break commit message parsing

We initially extracted PR info by regex-matching git log -1 --pretty=%s for the pattern Merge pull request #N from.... Worked fine until someone's repo used squash merge, which produces PR Title (#123) instead. Different regex entirely. We ended up needing three fallback strategies — extract from commit message, fall back to git log search, fall back to querying the GitHub API for the latest merged PR — before we had something reliable.

[skip ci] is a blunt instrument

We used [skip ci] in our automated version-bump commits to prevent infinite pipeline loops. Problem: [skip ci] skips every workflow on that commit — including the downstream QA/staging deploy pipelines when that commit gets merged forward. We had to invent our own convention ([skip version-bump]) and check for it explicitly in our Python scripts instead, rather than relying on GitHub's built-in skip behavior.

Race conditions in parallel merges

Two developers merging hotfixes minutes apart could both read the same stale VERSION file and both bump from 1.0.1, producing two commits both claiming to be 1.0.2. The fix was a GitHub Actions concurrency group with cancel-in-progress: false — this queues version bumps instead of running them in parallel, so the second one always reads the freshly-committed version from the first.

concurrency:
  group: version-bump-develop
  cancel-in-progress: false
Enter fullscreen mode Exit fullscreen mode

Branch protection blocks even your own automation

GITHUB_TOKEN respects branch protection rules — including rules that say "no direct pushes, PRs only." Our version-bump commits kept getting rejected. The fix was a Personal Access Token with elevated scope, used specifically for the automation's checkout step, plus explicit branch-protection bypass permissions for the github-actions[bot] actor.

VERSION file merge conflicts across branches

Every environment branch (develop, qa, staging, main) can end up with a slightly different VERSION file state depending on what's been promoted where. Merging develop → qa would frequently produce a conflict purely on that one file. We solved this with a .gitattributes merge strategy and, more reliably, a -X ours merge flag in our promotion scripts so the target branch's version always wins during a promotion — the version number isn't something that should be "merged," it's something that's authoritative per-branch.

What We'd Tell Anyone Starting This

Don't reinvent semantic versioning rules — but do decide who owns the version file. Every conflict we hit came back to ambiguity about which branch was the "source of truth" for a given version number.

Treat AI review as a second reviewer, not a replacement. Our policy: any critical finding from the AI must be resolved before a human approves. But it doesn't auto-block merges — humans still make the final call.

Build in idempotency everywhere. Pipelines get retried, PRs get merged twice by mistake, tags occasionally already exist. Every step in our pipeline now checks "does this already exist?" before creating anything.

GitHub's automatic behaviors ([skip ci], branch protection, squash merge formatting) will surprise you. Read the docs, but expect to discover the real behavior through production incidents anyway.

The Result

Every commit that lands in production today has been through: branch name validation, automated tests, an AI-reviewed diff with security and secret scanning, a human approval, an automatically generated version number, an AI-written release note, and — for anything touching main — three separate human sign-offs.

None of it required a person to remember to do any of those steps manually. That's the whole point.

If you're building something similar, happy to go deeper into any piece of this — the AI review prompt engineering, the version bump race-condition handling, or the emergency approval workflow. Drop a comment.

We run this pipeline on GitHub Actions with OpenAI's API for review and release-note generation. Happy to share specific workflow YAML if there's interest.

Top comments (0)