Darian Vance

Posted on Jan 6 • Originally published at wp.me

Solved: Bitbucket bait-and-switched, now charging $15/month per self-hosted runner

#devops #programming #tutorial #cloud

🚀 Executive Summary

TL;DR: Atlassian introduced an unexpected $15/month charge per self-hosted Bitbucket runner, shifting from a previously free model and significantly impacting CI/CD budgets. Organizations must now strategically re-evaluate their infrastructure by optimizing existing runners through dynamic scaling and pipeline improvements, migrating to Bitbucket Cloud’s managed runners, or exploring alternative CI/CD platforms like GitLab CI/CD or GitHub Actions which offer free self-hosted runner options.

🎯 Key Takeaways

Bitbucket’s new $15/month fee per self-hosted runner represents a fundamental shift in its CI/CD cost model, necessitating immediate re-evaluation of existing infrastructure and budgets.
Cost mitigation strategies for Bitbucket self-hosted runners include implementing dynamic scaling via cloud provider auto-scaling groups (ASG) or Kubernetes HPA/KEDA, leveraging larger instances, and optimizing pipeline execution through caching and splitting.
Alternative CI/CD platforms like GitLab CI/CD and GitHub Actions offer self-hosted runners without direct monthly fees, presenting viable migration paths for organizations seeking to avoid Bitbucket’s new charges while maintaining control over their build environments.

Organizations leveraging Bitbucket Pipelines for their CI/CD found themselves facing an unexpected cost increase as Atlassian began charging $15/month per self-hosted runner, shifting from a previously free model. This change necessitates a strategic re-evaluation of CI/CD infrastructure and cost optimization.

Symptoms: The Bitbucket Runner Bait-and-Switch

For years, Bitbucket users enjoyed the flexibility and cost-effectiveness of self-hosting their CI/CD runners, managing their compute infrastructure while benefiting from Bitbucket Pipelines orchestration. This model was particularly attractive for organizations with specific security requirements, on-premises resources, or high-volume builds where dedicated hardware could be more economical than managed cloud services.

The recent policy change by Atlassian, introducing a $15 monthly fee for each active self-hosted runner, has generated significant frustration among IT professionals. This isn’t merely an incremental price adjustment; it represents a fundamental shift in the cost model for a previously free component, catching many organizations off-guard and impacting their operational budgets.

Unexpected Cost Increase: Budget forecasting for CI/CD can be significantly disrupted, especially for teams running many concurrent or dedicated self-hosted runners.
Vendor Lock-in Perception: The sudden introduction of a fee for an essential, previously free service can lead to feelings of being “bait-and-switched,” making organizations wary of long-term commitment.
Operational Overhead vs. Cost: While self-hosting always implied operational overhead, the added monthly fee now forces a re-evaluation of whether that overhead, combined with the new charge, still offers value compared to managed alternatives or different platforms.

Facing this new reality, IT teams must assess their current setup and explore viable strategies to mitigate the impact.

Solution 1: Optimize Existing Bitbucket Self-Hosted Runners

If migrating away from Bitbucket Pipelines isn’t immediately feasible or desired, optimizing your current self-hosted runner infrastructure is the first step. The goal is to reduce the number of active runners required or to make each runner more cost-efficient.

Dynamically Scale Runners

Instead of maintaining a fixed number of runners that might be underutilized during off-peak hours and overloaded during peak times, implement dynamic scaling. This ensures you only pay for runners when they’re actively processing pipelines.

Cloud Provider Auto-Scaling Groups (ASG): For runners hosted on cloud VMs (e.g., AWS EC2, Azure VMs, GCP Compute Engine), use native auto-scaling groups. While Bitbucket Pipelines doesn’t directly expose queue metrics for direct scaling, you can use proxies:
- Monitor CPU/memory utilization of existing runner instances.
- Implement a custom solution that polls the Bitbucket Pipelines API for pending builds and adjusts ASG desired capacity accordingly.
- Use a scheduled scale-in during known idle periods (e.g., overnight, weekends).

Example AWS ASG User Data Script (Simplified):

  #!/bin/bash
  # Install AWS CLI, Docker, and other dependencies here

  # Fetch Bitbucket runner token from AWS Secrets Manager
  RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id YOUR_BITBUCKET_RUNNER_SECRET_ID --query SecretString --output text)
  WORKSPACE_ID="your-bitbucket-workspace-uuid"
  RUNNER_NAME_PREFIX="dynamic-runner-"

  # Download and extract Bitbucket Pipelines runner
  # This should ideally be pre-baked into an AMI for faster startup
  wget https://bitbucket.org/bitbucket-pipelines/atlassian-bitbucket-pipelines-runner/downloads/bitbucket-pipelines-runner.zip
  unzip bitbucket-pipelines-runner.zip -d atlassian-bitbucket-pipelines-runner
  chmod +x atlassian-bitbucket-pipelines-runner/bin/start.sh

  # Register and start the runner
  # Using hostname for a unique name
  atlassian-bitbucket-pipelines-runner/bin/start.sh \
      --token "$RUNNER_TOKEN" \
      --workspace "$WORKSPACE_ID" \
      --name "${RUNNER_NAME_PREFIX}$(hostname)" \
      --docker-privileged # Add any other required options

Kubernetes with Horizontal Pod Autoscaling (HPA) or KEDA: If your runners are containerized and deployed on Kubernetes, leverage HPA based on CPU/memory utilization. For more advanced queue-based scaling, KEDA (Kubernetes Event-Driven Autoscaling) can be used, potentially with a custom scaler that monitors Bitbucket Pipeline queues (if an exporter exists or can be built).

Example Kubernetes Deployment for a Bitbucket Runner:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: bitbucket-runner-deployment
    labels:
      app: bitbucket-runner
  spec:
    replicas: 1 # Start with a baseline, HPA will scale
    selector:
      matchLabels:
        app: bitbucket-runner
    template:
      metadata:
        labels:
          app: bitbucket-runner
      spec:
        containers:
        - name: runner
          image: your-custom-runner-image:latest # Image with runner binary, dependencies
          env:
          - name: BITBUCKET_RUNNER_TOKEN
            valueFrom:
              secretKeyRef:
                name: bitbucket-runner-secrets # Kubernetes secret
                key: token
          - name: BITBUCKET_WORKSPACE_ID
            value: "your-workspace-uuid"
          - name: BITBUCKET_RUNNER_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name # Use pod name for uniqueness
          # Add other runner specific environment variables
          volumeMounts:
          - name: docker-sock
            mountPath: /var/run/docker.sock
        volumes:
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock
  ---
  apiVersion: autoscaling/v2
  kind: HorizontalPodAutoscaler
  metadata:
    name: bitbucket-runner-hpa
  spec:
    scaleTargetRef:
      apiVersion: apps/v1
      kind: Deployment
      name: bitbucket-runner-deployment
    minReplicas: 1
    maxReplicas: 5 # Define your maximum capacity
    metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60 # Scale up when CPU exceeds 60%

Leverage Larger, More Capable Instances

Sometimes, fewer, more powerful runners can be more cost-effective than many small ones. A larger instance might complete jobs faster, leading to less overall execution time and potentially requiring fewer concurrent runners to meet demand. Evaluate your pipeline workloads:

Are they CPU-bound, memory-bound, or I/O-bound?
Can parallelism within a single runner (e.g., multiple Docker containers) be leveraged?
Consider the instance types offered by your cloud provider.

Optimize Pipeline Execution

Cache Dependencies: Use Bitbucket Pipelines caching mechanisms for build dependencies (npm, Maven, pip, etc.) to reduce build times.
Split Large Pipelines: Break down monolithic pipelines into smaller, more focused ones that can run independently or in parallel.
Docker Layer Caching: For Docker builds, optimize your Dockerfiles to leverage build cache effectively. Consider a private Docker registry for caching intermediate layers.
Cleanup Workspaces: Ensure build artifacts and temporary files are cleaned up at the end of each pipeline run to free up disk space and prevent performance degradation.

Solution 2: Migrate to Bitbucket Cloud Runners (Managed)

Atlassian offers managed cloud runners as an alternative to self-hosted. For many organizations, the shift might simplify operations and potentially reduce costs, especially for smaller teams or those with bursty, unpredictable workloads.

Cost Model Comparison

Bitbucket Cloud runners operate on a consumption-based model (build minutes), while self-hosted runners are now a fixed monthly fee per runner regardless of usage.

Free Tier: Standard plans include 2,500 build minutes per month. Premium plans get 3,500 minutes.
Paid Minutes: After the free tier, minutes are billed, typically around $10 for 1,000 minutes.
Self-Hosted Runner Cost: $15 per runner per month.

For example, if you run pipelines for 1,000 minutes a month and utilize only one self-hosted runner, the self-hosted option might appear cheaper. However, if that one runner is idle for a significant portion of the month, or if you need multiple runners for parallel execution, the managed option could become more economical due to its pay-per-use nature.

Pros and Cons


Feature	Bitbucket Self-Hosted Runners	Bitbucket Cloud Runners (Managed)
Cost Model	$15/month/runner	Free minutes (2,500/month), then ~ $10/1,000 minutes
Maintenance	High (OS, dependencies, scaling, security patching)	Zero (Atlassian manages infrastructure)
Customization	Full control (OS, specific software, hardware specs, network access)	Limited (standard Linux/Windows environments, pre-installed tools)
Security/Network	Runs within your private network, direct access to internal resources	Cloud-based (public internet access, VPC peering available for Enterprise plans)
Scaling	Manual or requires self-implemented autoscaling (e.g., ASG, Kubernetes)	Automatic (managed by Atlassian, scales seamlessly)
Initial Setup	Manual setup, runner registration, dependency installation	Zero setup, immediately available
Ideal Use Case	Strict compliance, custom environments, heavy workloads where self-hosted cost-per-minute is lower than managed	Standard workloads, lower overall build volume, preference for zero maintenance, fast setup

Solution 3: Explore Alternative CI/CD Platforms

If Bitbucket Pipelines, even with optimized self-hosted runners or by migrating to managed cloud runners, no longer meets your cost or feature expectations, it might be time to consider a complete migration to an alternative CI/CD platform. Several robust options offer flexible runner models.

GitLab CI/CD

GitLab offers a comprehensive, integrated DevOps platform, and its CI/CD capabilities are highly regarded. GitLab Runners can be self-hosted without additional fees for the runner itself (you only pay for your infrastructure costs), or you can use GitLab’s shared runners.

Pros: Deep integration if you also migrate your repositories to GitLab. Powerful, flexible YAML configuration. Self-hosted runners are free to deploy and manage. Strong community and feature set.
Cons: Requires migrating repositories and pipelines. Learning curve if unfamiliar with GitLab. Self-hosting GitLab itself can be complex for large instances.

Example .gitlab-ci.yml:

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  script:
    - echo "Building application..."
    - docker build -t myapp .
  tags:
    - my-self-hosted-docker-runner # Target a specific self-hosted runner

test_job:
  stage: test
  image: myapp:latest # Use the built image
  script:
    - echo "Running tests..."
    - npm test
  tags:
    - my-self-hosted-docker-runner

GitLab Runner Registration Command:

gitlab-runner register \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_PROJECT_OR_GROUP_RUNNER_TOKEN" \
  --executor "docker" \
  --description "My Self-Hosted Runner" \
  --tag-list "my-self-hosted-docker-runner" \
  --run-untagged="true" \
  --locked="false"

GitHub Actions

GitHub Actions has rapidly become a strong contender in the CI/CD space, especially for projects hosted on GitHub. Like GitLab, GitHub offers both managed runners (with a free tier and then pay-per-minute billing) and self-hosted runners, which are free.

Pros: Vast marketplace of pre-built actions. Excellent integration with GitHub repositories and ecosystem. Self-hosted runners are free. Strong community support.
Cons: Requires migrating repositories to GitHub. The YAML syntax can be verbose for complex workflows. Potential for vendor lock-in with GitHub-specific actions.

Example .github/workflows/main.yml:

name: CI/CD Pipeline

on: [push, pull_request]

jobs:
  build-and-test:
    runs-on: self-hosted # Use your self-hosted runner

    steps:
      - uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
      - name: Install dependencies
        run: npm ci
      - name: Run tests
        run: npm test

  deploy:
    needs: build-and-test
    runs-on: self-hosted
    steps:
      - name: Deploy to Staging
        run: echo "Deploying to staging environment..."

GitHub Self-Hosted Runner Setup (on your machine):

# Create a folder for the runner
mkdir actions-runner && cd actions-runner

# Download the latest runner package
# (Check GitHub Actions documentation for the latest version and OS)
curl -o actions-runner-linux-x64-2.317.0.tar.gz -L https://github.com/actions/runner/releases/download/v2.317.0/actions-runner-linux-x64-2.317.0.tar.gz

# Extract the installer
tar xzf ./actions-runner-linux-x64-2.317.0.tar.gz

# Configure the runner (follow prompts for repository URL and token)
./config.sh --url https://github.com/YOUR_ORG/YOUR_REPO --token YOUR_TOKEN

# Run the runner
./run.sh

Jenkins

Jenkins remains a highly popular, open-source automation server. While it requires significant self-management, its flexibility and vast plugin ecosystem are unmatched.

Pros: Completely free (only pay for infrastructure). Unlimited customization. Huge plugin ecosystem for integrations. Can run on virtually any platform. Ideal for complex, highly specific, or legacy environments.
Cons: High maintenance burden (Jenkins server, agents, upgrades, security). Steeper learning curve. UI can feel dated compared to newer platforms. Requires dedicated resources for the Jenkins controller.

Example Jenkinsfile (Declarative Pipeline):

pipeline {
    agent { label 'my-docker-agent' } // Target a specific Jenkins agent

    stages {
        stage('Build') {
            steps {
                script {
                    echo "Building application..."
                    sh 'docker build -t myapp .'
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    echo "Running tests..."
                    sh 'docker run myapp npm test'
                }
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying to staging...'
            }
        }
    }
}

Jenkins agents (runners) are connected via SSH, JNLP, or Kubernetes, and their setup depends on the chosen connection method and environment.

Comparison of CI/CD Platforms (Self-Hosted Runner Focus)


Feature	Bitbucket Pipelines (Self-Hosted)	GitLab CI/CD (Self-Hosted)	GitHub Actions (Self-Hosted)	Jenkins (Self-Hosted)
Runner Cost	$15/month/runner	Free (only pay for your infrastructure)	Free (only pay for your infrastructure)	Free (only pay for your infrastructure)
CI/CD Tool Cost	SaaS (part of Bitbucket plan)	Free (Community Edition) or paid tiers for GitLab instance	SaaS (part of GitHub plan)	Free (open source)
Integration with VCS	Tight with Bitbucket Repositories	Tight with GitLab Repositories	Tight with GitHub Repositories	Broad (plugins for Git, SVN, etc.)
Configuration	YAML (`bitbucket-pipelines.yml`)	YAML (`.gitlab-ci.yml`)	YAML (`.github/workflows/`)	Groovy (Declarative/Scripted Jenkinsfile)
Maintenance Burden (CI Tool)	Low (Atlassian manages the service)	High (if self-hosting GitLab server and runners)	Low (GitHub manages the service)	Very High (self-managing Jenkins controller and agents)
Ecosystem/Plugins	Growing (basic integrations, community steps)	Extensive (integrated features, community templates)	Vast (GitHub Marketplace actions)	Massive (thousands of plugins)
Learning Curve	Moderate	Moderate	Moderate	High
Ideal Use Case	Existing Bitbucket users, strict network isolation for builds, specific custom environments.	Teams consolidating VCS and CI/CD on GitLab, highly customizable CI at no runner cost.	Teams consolidating VCS and CI/CD on GitHub, leveraging rich marketplace, no runner cost.	Highly complex, legacy, on-premises, or unique CI/CD requirements, maximum control.

Conclusion

The introduction of a monthly fee for Bitbucket’s self-hosted runners is a significant change that demands a thoughtful response. Whether you choose to optimize your current setup, migrate to Bitbucket Cloud’s managed runners, or explore entirely new CI/CD platforms, the key is to perform a thorough cost-benefit analysis considering your specific organizational needs, technical capabilities, and long-term strategy. The right solution will balance cost-effectiveness with operational simplicity and the ability to meet your development velocity requirements.