DEV Community

Cover image for Multi-Repo to Monorepo: How I Automated 6 Go Microservice Releases and Then Made It 15x Faster
Amandeep Singh
Amandeep Singh Subscriber

Posted on

Multi-Repo to Monorepo: How I Automated 6 Go Microservice Releases and Then Made It 15x Faster

#go #devops #automation #monorepo

Last month I spent more than an hour cutting a release across six Go microservice repos. Tag log, wait for CI. Update sdk's go.mod to point at the new log SHA, push, wait for CI. Repeat for utils. Then do api, cli, and worker in parallel - except I forgot to bump cli's dependency and the build broke at 11pm.

That was the last manual release I did.

This is the story of automating that entire workflow with Jenkins + Python + GitLab, then realizing the multi-repo architecture was the real problem, and collapsing everything into a Go monorepo that's 15x faster at cutting releases.

The full setup runs on my laptop. You can fork it and try it yourself.

Table of Contents

The Six Modules

The project simulates a real production system with six Go modules that have strict dependency ordering:

Module Role Tag Scheme Depends On
log Logger (leaf, no deps) v0.x.0 -
sdk API client v0.x.0 log
utils Shared utilities v0.x.0 log, sdk
api/backend Backend APP-x.y.z log, utils
cli CLI cli-x.y.z log, sdk
worker Background v0.x.0 log, utils

The first three modules are sequential - sdk can't tag until log is tagged, utils can't tag until sdk is tagged. The last three are terminal - they can process in parallel once the sequential chain is done.

Every module lives on three long-lived branches: develop → release → master. A release means moving code through all three, in all six repos, in the right order.

That's the problem. Do it manually and you're juggling 6 repos × 3 branches × dependency ordering. One forgotten go mod tidy and you're debugging at midnight.

The Stack

Everything runs on a MacBook. No cloud CI, no SaaS - just local tools wired together.


 MacBook                                         GitLab.com
+-------------------+    ngrok tunnel    +------------------------+
|  Jenkins LTS      | <===============> |  Webhooks (push / MR)  |
|  (brew service)   |                   |  Commit status API     |
|  :8080            | ================> |  6 repos (multi-repo)  |
|                   |   git push,       |  - or -                |
|  Python 3 + Go    |   API calls       |  1 repo  (monorepo)    |
+-------------------+                   +------------------------+
Enter fullscreen mode Exit fullscreen mode

The key pieces:

  • Jenkins LTS via brew install jenkins-lts, running as a launched service
  • ngrok exposing localhost:8080 → https://enclosure-version-moocher.ngrok-free.dev/ so GitLab integration/webhook can reach Jenkins
  • GitLab.com hosting the repos
  • Python (python-gitlab + GitPython) for the automation engine
  • Go 1.24 with workspace support (go.work)

Jenkins ↔ GitLab Integration

Configured using GitLab’s native Jenkins Integration combined with ngrok for local tunneling:

  1. GitLab → Jenkins: Native integration automatically triggers local Jenkins pipelines on push and MR events.
  2. Jenkins → GitLab: Reports build progress and status back via API, displaying green/red status checks directly on commits and merge requests.

Phase 1: Multi-Repo Automation

The Setup: 6 Repos, 7 Jenkins Jobs

In the multi-repo world, each module is its own GitLab repo under the multirepo-release/ group:

multirepo-release/
  ├── log/          → gitlab.com:multirepo-release/log.git
  ├── sdk/          → gitlab.com:multirepo-release/sdk.git
  ├── utils/        → gitlab.com:multirepo-release/utils.git
  ├── api/          → gitlab.com:multirepo-release/api.git
  ├── cli/          → gitlab.com:multirepo-release/cli.git
  ├── worker/       → gitlab.com:multirepo-release/worker.git
  └── pipelines/    → gitlab.com:multirepo-release/pipelines.git
Enter fullscreen mode Exit fullscreen mode

Each repo gets its own Jenkins CI job (6 jobs), plus one orchestration job that runs the automation Python script. That's 7 Jenkins jobs to maintain.

The Automation Engine: 1,800 Lines of Python

The pipelines/ repo contains the brain: a Python script (poc_automation.py) that talks to the GitLab API and performs git operations. One Jenkins job, five actions:

Action What it does
release_cut Merges develop → release for all 6 repos, in dependency order
backmerge Merges release → develop (or master → release)
rc_create Merges release → master, tags each module, bumps go.mod versions
version_bump Bumps version constants in api and cli on develop
compute_versions Calculates next semver tags based on existing tags

How a Release Cut Actually Works

When you trigger release_cut, the script:

  1. Sequential phase - processes log, sdk, utils one at a time:

    • Creates an intermediate branch from release
    • Merges develop into it (handling go.mod conflicts automatically)
    • Creates a GitLab MR
    • Waits for approval (or auto-merges if configured)
    • Records the merged SHA - the next repo needs it for go mod tidy

  2. Parallel phase - processes api, cli, worker simultaneously:

    • Same merge flow, but all three run concurrently since they're terminal nodes

The hardest part: cross-repo dependency tracking. When log merges to release, sdk needs to run go get gitlab.com/multirepo-release/log@<new-SHA> to pin to the exact commit. The script tracks these SHAs in self.merged_target_shas and passes them forward through the chain.

Conflict Handling

When code conflicts happen (not just go.mod), the pipeline doesn't crash. It:

  1. Creates a conflict branch cut from the source (so GitLab renders the real diff)
  2. Opens an MR with [CONFLICT] in the title
  3. Polls every 30 seconds for manual resolution + approval
  4. Auto-merges once both conditions are met
  5. Continues the pipeline

The HTML Report

Every automation run generates a styled HTML report archived in Jenkins. It logs every action taken: branches created, MRs opened, approvals received, tags applied.

Phase 2: The Monorepo Pivot

The multi-repo automation worked. But maintaining it was its own job:

  • 7 Jenkins jobs to configure and monitor
  • 6 Jenkinsfile.ci files (103 lines each, identical except for the module name)
  • 1,800 lines of Python managing 6 separate GitLab projects, tracking SHAs across repos, handling per-repo cloning
  • 6 MRs per release cut - one per repo, each needing approval

The question became: what if all six modules lived in one repo?

go.work: The Go Workspace

Go 1.18 introduced workspaces (go.work), which let multiple modules in a single repo resolve each other locally during development while still being independently importable by external consumers.

// go.work - the workspace file at the repo root
go 1.26

use (
    ./api/backend
    ./cli
    ./log
    ./sdk
    ./utils
    ./worker
)

replace (
    gitlab.com/singhamandeep007-group/monorepo/log v0.3.0 => ./log
    gitlab.com/singhamandeep007-group/monorepo/sdk v0.3.0 => ./sdk
    gitlab.com/singhamandeep007-group/monorepo/utils v0.3.0 => ./utils
)
Enter fullscreen mode Exit fullscreen mode

The replace directives are the key insight: during local development and CI, Go resolves internal dependencies from the local directory. But when an external project runs go get gitlab.com/.../monorepo/log@v0.3.0, Go fetches the tagged version from GitLab. Best of both worlds.

Per-Module Tagging in a Monorepo

Go's module system supports directory-prefixed tags. In a monorepo, each module gets its own tag with a path prefix:

# Each module tagged independently, on the same commit
git tag log/v0.3.0
git tag sdk/v0.3.0
git tag utils/v0.3.0
git tag api/backend/APP-1.3.0
git tag cli/cli-1.3.0
git tag worker/v0.3.0
Enter fullscreen mode Exit fullscreen mode

When someone runs go get gitlab.com/singhamandeep007-group/monorepo/sdk@v0.3.0, Go looks for the tag sdk/v0.3.0 - the module path prefix matches the tag prefix. Each module can version independently while living in the same repo.

What Simplified

The monorepo automation (monorepo-pipelines/poc_automation.py) dropped from 1,762 to 1,143 lines - a 35% reduction. Here's what went away:

Multi-repo complexity Monorepo equivalent
6 separate GitLab project objects 1 project object
Clone + manage 6 local git repos No local cloning needed
Track merged_target_shas across repos Not needed - same repo, same SHA
go get module@SHA + go mod tidy per repo GitLab file API to edit go.mod directly
6 MRs per release cut 1 MR
sync_dependencies action Not needed

Third-Party Consumption

An external project can import any module from the monorepo individually:

// third-party project's go.mod
module gitlab.com/singhamandeep007-group

go 1.26

require gitlab.com/singhamandeep007-group/monorepo/log v0.3.0
Enter fullscreen mode Exit fullscreen mode 
// main.go
package main

import "gitlab.com/singhamandeep007-group/monorepo/log"

func main() {
    logger := log.NewLogger()
    logger.Info("Using log from the monorepo")
}
Enter fullscreen mode Exit fullscreen mode

The consumer doesn't need the entire monorepo - Go fetches only the log module at the tagged version.

The Unified CI Pipeline

One Pipeline, All Modules, In Parallel

The monorepo CI (jenkins/ci.Jenkinsfile, 324 lines) replaces 6 identical per-repo Jenkinsfile.ci files (636 lines total). It fans out all six modules in parallel:

For each module, in parallel:

  1. go vet ./...
  2. go test -count=1 -race -covermode=atomic → JUnit XML + coverage HTML
  3. govulncheck → JSON + human-readable
  4. go build (only for api/backend and cli) → multi-arch binaries + SHA256SUMS 
stage("All Packages") {
    parallel pkgStages  // 6 modules run simultaneously
}
Enter fullscreen mode Exit fullscreen mode

All six run with failFast: false - a failure in one module doesn't mask problems in others. The pipeline produces an aggregated ci-report.html showing every module's test results, coverage percentage, and vulnerability count in one page.

Multi-Arch Builds

Terminal binaries (api/backend, cli) build for three targets:

# CGO_ENABLED=0 for cross-compilation (no C dependencies)
GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 go build -o bin/api-darwin-arm64
GOOS=linux  GOARCH=amd64 CGO_ENABLED=0 go build -o bin/api-linux-amd64
GOOS=linux  GOARCH=arm64 CGO_ENABLED=0 go build -o bin/api-linux-arm64

# SHA256 checksums for verification
shasum -a 256 bin/* > bin/SHA256SUMS
Enter fullscreen mode Exit fullscreen mode

The Release Automation Pipeline

The release automation lives in a separate repo (monorepo-pipelines) - keeping release tooling isolated from the application code. Same five actions as before, but operating on a single repo:

After an RC, the script tags each module and updates go.mod + go.work via the GitLab file API - no local cloning needed.

Real Numbers

These come from 451 multi-repo builds and 36 monorepo builds on the same Jenkins instance, running the same Go modules.

Release Automation Speed

Action Multi-Repo (avg) Monorepo (avg) Speedup
release_cut 1,170s (~19 min) 75s (~1 min) 15.5x
backmerge 688s (~11 min) 35s 19.4x
rc_create 501s (~8 min) 73s (~1 min) 6.9x

The multi-repo release_cut takes ~19 minutes because it processes 6 repos sequentially (3 sequential + 3 parallel), creating 6 MRs, waiting for each CI to pass, and tracking SHAs across repos. The monorepo version creates 1 MR and is done in ~75 seconds.

CI Pipeline

Metric Multi-Repo Monorepo
Jenkins jobs 7 (6 CI + 1 orchestration) 2 (1 CI + 1 automation)
CI wall time ~132s (sequential across 6 jobs) ~55s (parallel in 1 job)
Jenkinsfile maintenance 6 × 103 lines (identical) 1 × 324 lines
MRs per release cut 6 1

Codebase

Metric Multi-Repo Monorepo Change
Automation Python 1,800 lines 1,143 lines -36%
CI pipeline config 618 lines (6 files) 324 lines (1 file) -48%
Jenkins orchestration 274 lines 231 lines -16%
Total pipeline code 2,692 lines 1,698 lines -37%

Total Build History

Multi-Repo Monorepo
CI builds 385 24
Automation builds 66 12
Total builds 451 36
Jenkins jobs to maintain 7 2

Caveats and Gotchas

A few things that bit me along the way - saving you the debugging time:

go.work replace versions must match go.mod exactly

If sdk/go.mod says require log v0.5.0 but go.work has replace log v0.3.0 => ./log, Go ignores the replace and tries to fetch log v0.5.0 from the remote. The versions must match exactly. This caused CI failures on all three branches when our RC automation bumped go.mod versions without creating matching tags.

Jenkins doesn't load shell profiles

If you install Go via asdf or a version manager, Jenkins won't find it - it runs a non-login, non-interactive shell. Set GOROOT and PATH explicitly in the pipeline environment:

environment {
    GOROOT = '/usr/local/go'
    PATH   = "/usr/local/go/bin:${env.HOME}/go/bin:${env.PATH}"
}
Enter fullscreen mode Exit fullscreen mode

Private modules need three environment variables

For private GitLab-hosted Go modules, GOPRIVATE alone isn't enough. You also need GONOSUMCHECK and GONOSUMDB to prevent checksum database lookups that will fail:

GOPRIVATE    = 'gitlab.com/your-group/*'
GONOSUMCHECK = 'gitlab.com/your-group/*'
GONOSUMDB    = 'gitlab.com/your-group/*'
Enter fullscreen mode Exit fullscreen mode

CGO and cross-compilation don't mix (on macOS)

CGO_ENABLED=1 is needed for -race in tests, but cross-compiling to linux/amd64 on macOS with CGO on fails because there's no Linux C toolchain. Solution: CGO_ENABLED=1 globally for tests, CGO_ENABLED=0 per build command:

# Tests: CGO on (for -race)
CGO_ENABLED=1 go test -race ./...

# Builds: CGO off (for cross-compilation)
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o bin/api-linux-amd64
Enter fullscreen mode Exit fullscreen mode

Groovy triple-quoted strings mangle Python escape sequences

A \n inside a Groovy """...""" string becomes a literal newline before the embedded Python sees it, causing SyntaxError: EOL while scanning string literal. Use chr(10) and chr(9) instead:

# Breaks: Groovy converts \n to a real newline
line.startswith("\nFAIL\t")

# Works: chr() bypasses Groovy's string processing
line.startswith(chr(10) + "FAIL" + chr(9))
Enter fullscreen mode Exit fullscreen mode

Try It Yourself

Both repos are on GitLab:

  • Monorepo (Go source + CI): gitlab.com/singhamandeep007-group/monorepo
  • Monorepo Pipelines (release automation): gitlab.com/singhamandeep007-group/monorepo-pipelines
  • Multi-repo version (for comparison): gitlab.com/multirepo-release/

Quick Start 

# 1. Install Jenkins
brew install jenkins-lts
brew services start jenkins-lts

# 2. Install ngrok and start tunnel
ngrok http 8080 --url your-subdomain.ngrok-free.dev

# 3. Clone and explore
git clone git@gitlab.com:singhamandeep007-group/monorepo.git
cd monorepo
go work sync
make test-all
Enter fullscreen mode Exit fullscreen mode

What's Next

This is a POC - production would need:

  • Secrets management: rotate GitLab tokens, use Jenkins credential rotation
  • Agent scaling: Jenkins agents instead of running everything on the controller
  • Notification: Slack/email integration for merge approvals and failures
  • Caching: Go module cache persistence across builds

Top comments (0)