AWS Building Blocks for a Cost-Effective Self-Hosted Azure DevOps Runner Fleet

Self-hosted CI runners give you three main benefits that managed runners don’t; direct access to private VPC resources, the ability to customize the build machines with your own tools, and better control over costs. The trade-off is that you have to manage the machines yourself, but with a few AWS services, most of that management can be automated.
This article explains the key AWS components used to build a runner fleet that is fast, cost-effective, and doesn’t require managing long-lived credentials.

Four AWS features make this possible:

1. EC2 Image Builder: reproducible golden AMIs
2. Auto Scaling Groups: a self-healing fleet
3. ASG Scheduled Actions: schedule-based cost control
4. IAM OIDC federation: short-lived credentials with no stored keys

We’ll use Azure DevOps agents as an example, but the same AWS approach can be used for other self-hosted runners, such as GitHub Actions, GitLab CI, Jenkins etc.

Architecture at a glance

Prerequisites

• An AWS account with a VPC that has private subnets and outbound internet (a NAT gateway, or VPC endpoints if you want to stay fully private). Agents need to reach dev.azure.com.
• An Azure DevOps organization and project, and permission to create an agent pool.
• Basic familiarity with the AWS Console or Terraform. I'll show illustrative snippets; adapt to your IaC tool of choice.

A quick vocabulary check for beginners:

• Self-hosted agent: a machine running the Azure Pipelines agent software, registered to a pool. Pipelines targeting that pool run on your machines.
• Golden AMI: a pre-built machine image with all your tools already installed. Boot from it and you're ready in seconds.
• Auto Scaling Group: AWS keeps N identical instances running from a launch template, replacing any that die.
• OIDC federation: a trust relationship that lets your pipeline prove its identity and receive temporary AWS credentials, instead of you pasting an access key into a settings box.

Two of these; "Image Builder" and "OIDC" are AWS features many teams have never touched, so we'll spend the most time there.

Step 1: Bake a golden AMI with EC2 Image Builder

Installing tools at boot (via a long user-data script) is slow and fragile, a package mirror hiccup fails every new instance. Instead, install everything once into an image.

EC2 Image Builder has three pieces:

• Component: a recipe of install steps (YAML).
• Image recipe: a base AMI (e.g. Amazon Linux 2023) and your components.
• Pipeline: builds, tests, and outputs a new AMI version.

A minimal component that installs the Azure DevOps agent's dependencies plus common CI tools:

name: ci-agent-tools
schemaVersion: 1.0
phases:
  - name: build
    steps:
      - name: install
        action: ExecuteBash
        inputs:
          commands:
            - dnf install -y git jq unzip tar
            # container builds
            - dnf install -y docker && systemctl enable docker
            # AWS CLI v2
            - curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscli.zip
            - unzip -q /tmp/awscli.zip -d /tmp && /tmp/aws/install
            # download the Azure Pipelines agent (pin the version!)
            - mkdir -p /opt/azagent && cd /opt/azagent
            - curl -sL https://vstsagentpackage.azureedge.net/agent/3.240.0/vsts-agent-linux-x64-3.240.0.tar.gz | tar xz
            - ./bin/installdependencies.sh

Tip: pin your versions. "Latest" makes every rebuild non-reproducible and can break a Friday-afternoon build with a surprise upgrade. Bump versions deliberately.

Point an Image Builder pipeline at your recipe, run it, and you get an AMI ID. Image Builder can also run tests and copy the AMI to other regions. Image versions are immutable, to change tooling, bump the component version and rebuild.

Step 2: A launch template that self-registers agents

The AMI has the agent binary; a fresh instance still has to register with your Azure DevOps pool on boot and deregister on shutdown. That's the launch template's user-data.

Registration needs a credential. Don't use a PAT, it's tied to a person (their expiry, their access, breaks when they leave). Use a Microsoft Entra ID service principal: a non-personal identity you grant Agent Pools (Read & manage), authenticated with --auth SP. The instance pulls its secret from Secrets Manager via its IAM instance profile, nothing sensitive is baked into the AMI.

resource "aws_launch_template" "runner" {
  image_id      = var.golden_ami_id
  instance_type = "t3.large"
  iam_instance_profile { name = aws_iam_instance_profile.runner.name }  # GetSecretValue on ONE secret

  user_data = base64encode(templatefile("${path.module}/user-data.sh.tftpl", {
    azdo_org      = "https://dev.azure.com/your-org"
    agent_pool    = "aws-linux-fleet"
    sp_secret_arn = aws_secretsmanager_secret.runner_sp.arn
    sp_client_id  = var.sp_client_id   # Entra app ID- not a secret
    sp_tenant_id  = var.sp_tenant_id
  }))
}

#!/usr/bin/env bash
set -euo pipefail
cd /opt/azdo-agent

# Fetch the SP secret using the instance profile, nothing baked into the AMI.
SP_SECRET="$(aws secretsmanager get-secret-value --secret-id "${sp_secret_arn}" \
  --query SecretString --output text)"

# Register as the service principal (NOT a PAT).
./config.sh --unattended --url "${azdo_org}" \
  --auth SP --clientid "${sp_client_id}" --clientsecret "$SP_SECRET" --tenantid "${sp_tenant_id}" \
  --pool "${agent_pool}" --agent "runner-$(hostname)" --replace --acceptTeeEula

./svc.sh install && ./svc.sh start   # add --once via run.sh instead for ephemeral one-job agents

Step 3: Wrap it in an Auto Scaling Group

The ASG turns your launch template into a managed fleet: it keeps desired instances running, replaces unhealthy ones, and spreads them across availability zones.

Illustrative Terraform:

resource "aws_autoscaling_group" "runners" {
  name                = "azdo-runners"
  min_size            = 1
  max_size            = 5
  desired_capacity    = 2
  vpc_zone_identifier = var.private_subnet_ids   # 2+ AZs

  launch_template {
    id      = aws_launch_template.runner.id
    version = "$Latest"
  }

  # let a running build finish before an instance is removed
  initial_lifecycle_hook {
    name                 = "graceful-drain"
    lifecycle_transition = "autoscaling:EC2_INSTANCE_TERMINATING"
    default_result       = "CONTINUE"
    heartbeat_timeout    = 900
  }
}

Now you have a self-healing fleet. Push the desired capacity up when you need more parallelism; the ASG handles the rest.

Step 4: Scale down nights and weekends (cost savings)

Most teams don't build at 3 a.m. on a Sunday, yet a static fleet bills 24/7. ASG scheduled actions let you shrink the fleet off-hours and grow it back for the workday. Cron is in UTC, mind your timezone.

# Friday 20:00 UTC wind down to a single warm agent for the weekend
resource "aws_autoscaling_schedule" "weekend_down" {
  scheduled_action_name  = "weekend-down"
  autoscaling_group_name = aws_autoscaling_group.runners.name
  recurrence             = "0 20 * * FRI"
  min_size               = 1
  max_size               = 5
  desired_capacity       = 1
}

# Monday 06:00 UTC, back to full capacity for the week
resource "aws_autoscaling_schedule" "weekday_up" {
  scheduled_action_name  = "weekday-up"
  autoscaling_group_name = aws_autoscaling_group.runners.name
  recurrence             = "0 6 * * MON"
  min_size               = 2
  max_size               = 5
  desired_capacity       = 2
}

Rough intuition on savings: a fleet that runs at full size ~50 hours a week (business hours) instead of 168 is roughly a 70% compute cost cut versus always-on, before you even add Spot instances. (For non-critical builds, Spot in the launch template saves more; just handle interruptions with the same graceful-drain hook.)

Step 5: Keyless AWS access with OIDC federation

Here's the part that quietly de-risks your whole setup. The classic way to let a pipeline call AWS is to create an IAM user, generate an access key, and paste it into an Azure DevOps service connection. That key is long-lived, sits in your CI settings, and is a prime target if it leaks.

OIDC (workload identity federation) removes the key entirely. Instead:

• Azure DevOps mints a short-lived OIDC token describing the pipeline's identity.
• AWS trusts Azure DevOps as an identity provider.
• Your pipeline assumes an IAM role, receiving temporary credentials that expire in an hour.

Nothing to store, nothing to rotate, nothing to leak.
On the AWS side, create an IAM OIDC identity provider for Azure DevOps and a role that trusts it:

resource "aws_iam_openid_connect_provider" "azdo" {
  url             = "https://vstoken.dev.azure.com/<your-org-guid>"
  client_id_list  = ["api://AzureADTokenExchange"]
  thumbprint_list = ["<oidc-endpoint-thumbprint>"]
}

resource "aws_iam_role" "pipeline" {
  name = "azdo-pipeline"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Federated = aws_iam_openid_connect_provider.azdo.arn }
      Action    = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "vstoken.dev.azure.com/<your-org-guid>:aud" = "api://AzureADTokenExchange"
          # lock it to ONE service connection: sc://<org>/<project>/<connection-name>
          "vstoken.dev.azure.com/<your-org-guid>:sub" = "sc://your-org/your-project/aws-oidc"
        }
      }
    }]
  })
}

Tip: Why the sub condition matters: without pinning sub to your exact service connection, any Azure DevOps project that can reach your org could assume the role. The sub claim (sc://org/project/connection) is your lock. Always scope it.

On the Azure DevOps side, create an AWS service connection using Workload Identity Federation (not access keys), and point it at the role ARN. Azure DevOps generates the issuer/subject values you plug into the trust policy above. From then on, tasks like the AWS CLI or AWS Shell Script task authenticate by assuming the role, no keys anywhere.

Two roles, two jobs. Keep these separate: the instance profile on the runner (what the machine can do e.g. read the agent secrets from Secrets Manager) and the OIDC pipeline role (what a build can do in AWS). Least privilege on each.

Common beginner gotchas

• Agents can't reach Azure DevOps - they need outbound 443 to dev.azure.com / *.vsassets.io. Via NAT gateway, or lock it down with an egress allowlist/proxy.
• Offline agents piling up - you skipped deregistration on termination. Add the config.sh remove step.
• Changing tooling doesn't take - AMIs are immutable; bump the Image Builder version and refresh the ASG (an instance refresh rolls the fleet onto the new AMI).
• Cron fired at the wrong time - ASG schedules are UTC.
• Builds die mid-run on scale-in - add the graceful-drain lifecycle hook.

Supporting AWS services

• Secrets Manager: store the runner's registration token here; the instance fetches it at boot via its instance profile. Never bake secrets into an AMI or launch template.
• VPC endpoints / NAT: runners in private subnets reach AWS APIs via endpoints (private, cheaper egress) and the public internet via NAT only where needed.
• CloudWatch: ASG and instance metrics for fleet health and right-sizing.