Killing Jenkins Was the Best Career Move I Made This Year

Senior DevOps Engineer | 15 microservices | 2,800 deployments last month

Let me tell you about the day I took down our Jenkins master during peak traffic. 3:47 PM on a Thursday. 147 builds queued. 12 angry devs in Slack. VP of Engineering asking "what's the ETA?" like I had a crystal ball.

I fixed it by disabling the master node for 14 minutes while the database recovered. Then I spent the weekend migrating everything to GitHub Actions. That was 8 months ago. We haven't had a CI outage since.

Here's what I learned, what broke, and what I'd do differently.

The Problem Wasn't Jenkins. It Was Us.
Jenkins worked fine for 4 years. We had:

15 microservices with 200+ test suites each

6 shared Jenkins agents in Kubernetes

3,400 lines of pipeline-as-code

87 plugins (most of them obsolete)

1 overworked master node with 16GB RAM

The crash happened because a developer pushed a PR that triggered 27 downstream jobs simultaneously. The master ran out of heap space. Not because of a memory leak—because we never tuned -Xmx past 8GB.

What broke: We had 4 separate Jenkinsfiles per repo (build, test, deploy, rollback). Devs kept copying them from Stack Overflow without understanding the Groovy syntax. Pipeline logs were 40MB of noise.

How I fixed it: I didn't. I killed it.

Why GitHub Actions Won (For Us)
Metric Jenkins GitHub Actions
Build time (avg) 8m 42s 3m 17s
Pipeline failures Dev: "works on my machine" Dev: sees exact error in PR
Plugins to maintain 87 13 (all official)
Cost/month $427 (EC2) $0 (free tier + 2,000 minutes)
Developer complaints Daily Zero in 8 months
The real win: devs debug their own failures now. No more "hey DevOps, my pipeline is red" tickets. They see the exact step that failed, with the exact YAML line, right in the PR checks tab.

No more SSH-ing into Jenkins agents to read workspace logs. No more "can you restart the master?" at 2 AM.

The Migration: What Actually Worked
Phase 1: The Parallel Run (2 weeks)
We kept Jenkins as primary while I ported the simplest service to Actions:

yaml
name: CI
on: [push, pull_request]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm test
- run: npm run build
That's it. 14 lines. The Jenkinsfile for the same service was 187 lines.

Phase 2: The Matrix Strategy (1 week)
Our biggest headache in Jenkins was running tests across 6 Node versions and 3 databases. Actions handled this natively:

yaml
test-matrix:
runs-on: ubuntu-latest
strategy:
matrix:
node: [16, 18, 20]
db: [postgres, mysql, sqlite]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
- run: docker-compose up -d ${{ matrix.db }}
- run: npm test
18 parallel jobs. 4 minutes total. Jenkins took 12 minutes sequentially because we never configured parallel stages properly.

Phase 3: The Monorepo Migration (The Painful One)
We have 3 microservices sharing code in one repo. Jenkins handled this with a custom Groovy script that conditionally built based on changed paths.

Actions does this natively with path filters:

yaml
on:
push:
paths:
- 'services/auth/'
- 'shared/'
pull_request:
paths:
- 'services/auth/'
- 'shared/'
But we needed to build and deploy only the changed service. Here's the pattern that works:

yaml
changed-files:
uses: actions/checkout@v4
with:
fetch-depth: 2

• id: files
  uses: tj-actions/changed-files@v39
  with:
  files: |
  services/auth/**

• name: build-auth
  if: steps.files.outputs.any_changed == 'true'
  run: |
  cd services/auth
  docker build -t auth-service .
  docker push ...
  What I'd do differently: I'd split the monorepo sooner. We wasted 3 days fighting the path filter syntax. Use tj-actions/changed-files from day one. It's the only third-party action I recommend.

The Security Nightmare We Fixed
In Jenkins, we stored secrets as environment variables in the master's configuration. Every developer with "read" access could see them in the build logs if they added printenv.

In Actions, we use GitHub Secrets scoped to the repository:

yaml

• name: Deploy to ECS env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} ECR_REPO: ${{ secrets.ECR_REPO }} run: | aws ecs update-service ... You can also use environment-level secrets for staging vs production:

yaml
deploy-prod:
runs-on: ubuntu-latest
environment: production
steps:
- run: deploy.sh
env:
API_KEY: ${{ secrets.PROD_API_KEY }}
The one gotcha: You can't reuse the same secret name across environments. secrets.API_KEY exists once. Name them secrets.DEV_API_KEY, secrets.PROD_API_KEY.

The Things That Almost Broke Us

1. OIDC vs Static Credentials We started with static AWS keys rotated monthly. Then a developer committed dev.env with credentials. Never again.

Use OIDC authentication:

yaml

• name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT }}:role/github-actions aws-region: us-east-1 No keys. No rotation. Each run gets a temporary session.

What broke: I forgot to update the trust relationship when we added a new repo. 5 AM Sunday, deploy failed, I'm reading AWS docs on my phone.

The fix: Use a reusable workflow with the OIDC config baked in:

yaml

.github/workflows/deploy.yml (shared)

on:
workflow_call:
inputs:
environment:
required: true
type: string

jobs:
deploy:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT }}:role/github-actions-${{ inputs.environment }}
Now every repo calls this. The trust relationship is in one place.

1. Docker Build Cache Our Jenkins agents stored Docker layers in /var/lib/docker. Builds were fast because images were cached locally.

GitHub Actions runners are ephemeral. No cache = 6-minute builds from scratch.

The fix: Use GitHub's cache action for Docker layers:

yaml

• name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

• name: Cache Docker layers
  uses: actions/cache@v3
  with:
  path: /tmp/.buildx-cache
  key: ${{ runner.os }}-buildx-${{ github.sha }}
  restore-keys: |
  ${{ runner.os }}-buildx-

• name: Build and push
  uses: docker/build-push-action@v5
  with:
  cache-from: type=local,src=/tmp/.buildx-cache
  cache-to: type=local,dest=/tmp/.buildx-cache
  Build time dropped from 6m to 1m 40s. The cache key uses the commit SHA, so PRs restore from main's cache.

1. Self-Hosted Runners (The "We're Too Big" Problem) We peaked at 2,800 runs/month. GitHub-hosted runners were costing us $0.008/minute × ~1,200 minutes/day = $288/day.

We moved to self-hosted runners on AWS EC2 Spot instances. Cost dropped to $0.003/minute.

The YAML:

yaml
runs-on: self-hosted
The catch: Self-hosted runners don't auto-update. We're running Ubuntu 22.04 LTS with manual patching. I have an Ansible playbook that rebuilds the AMI weekly.

The setup (Terraform):

hcl
resource "aws_ec2_instance" "runner" {
ami = data.aws_ami.ubuntu.id
instance_type = "c6i.4xlarge"
spot_price = "0.15"

user_data = file("${path.module}/user-data.sh")

tags = {
Name = "github-actions-runner-${count.index}"
}
}
We have 6 runners. 3 for PRs, 3 for main branch. The PR runners terminate after 30 minutes idle. Main runners stay hot.

What I'd do differently: Start with GitHub-hosted until you actually need self-hosted. We prematurely optimized. The $288/day was worth the zero maintenance in hindsight.

What Broke That I Didn't Expect

1. The GITHUB_TOKEN Permissions Issue Our CD pipeline needed to comment on PRs after deployment. I used the default ${{ secrets.GITHUB_TOKEN }} and got a 403.

Solution: Set permissions at the job level:

yaml
jobs:
deploy:
runs-on: self-hosted
permissions:
contents: read
pull-requests: write
issues: write
steps:
- name: Comment on PR
uses: actions/github-script@v7
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '🚀 Deployed to production'
})
Took me 3 hours of trial and error. The GitHub docs bury this in the "Advanced" section.

1. Concurrency Limits Two PRs trying to deploy to the same environment simultaneously. Database migrations conflict. The first succeeds, the second fails with "table already exists."

The fix: Use concurrency groups:

yaml
concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}
cancel-in-progress: true
Now the second job waits for the first to finish. No more migration conflicts.

1. The 60-Minute Timeout One of our E2E test suites takes 45 minutes on a good day. GitHub-hosted runners have a 6-hour limit. Self-hosted don't (we set our own).

But we had a test that hung for 2 hours because of a race condition. The runner stayed active, blocking the queue.

Fix: Set explicit timeouts:

yaml
jobs:
e2e:
runs-on: self-hosted
timeout-minutes: 55
steps:
- run: npm run test:e2e
Now it fails at 55 minutes, not 120. The race condition got fixed because developers actually saw the failure.

The Numbers After 8 Months
Metric Before After
Average build time 8m 42s 3m 17s
Deployment frequency 42/day 93/day
CI failure rate 8% 1.2%
Pipeline code (total lines) 3,400 620
Dev tickets to DevOps 34/week 3/week
Sleep score (my Oura ring) 67 83
The last one is real. No more 3 AM Jenkins alerts. No more "can you restart the agent?" at 11 PM.

What I'd Do Differently (The Honest Retro)
Migrate one service at a time. I did the monorepo last. Should've done it first. The monorepo had the most complexity and cost us the most time.

Start with reusable workflows. I wrote the same YAML 15 times. Now I have a central .github/workflows/reusable-test.yml that all repos import:

yaml
name: Test
on: [workflow_call]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm test
Don't try to replicate Jenkins 1:1. I wasted a week making Actions "look like Jenkins." The pipeline should match your workflow, not your old tool.

Document the failure points. When I migrated, I kept notes on every error. That became our internal wiki. Now new DevOps engineers onboard in 2 days instead of 2 weeks.

Set up monitoring before the migration. I had no baseline for build times. I assumed Actions was faster. It was, but I couldn't prove it. Use the GitHub API to log metrics from day one.

The Bottom Line
Jenkins is a fine tool if you have 1-2 services and a dedicated team to maintain it. We had 15 services and me.

GitHub Actions eliminated the maintenance overhead. Developers fix their own pipelines because the errors are in their PRs, not buried in a Jenkins build #4837 log.

The migration cost me 3 weeks of focused work. The ROI was 2 months.

Would I do it again? In a heartbeat. But I'd start the monorepo split 6 months earlier and I'd write reusable workflows from day one.

My advice if you're considering it: Try the migration on a Sunday. Pick your simplest service. Get it running in Actions with the PR check passing. See how many lines of YAML it takes. Compare that to your Jenkinsfile. Then decide.

Just don't kill your Jenkins master at 3:47 PM on a Thursday. Learn from me.

Senior DevOps Engineer. 15 microservices. 2,800 deployments/month. 0 CI outages in 8 months.

Connect if you want to compare war stories. I've got more.