DEV Community: Ederson Brilhante

ForgeMT: GitHub Actions at Scale with Security and Multi-Tenancy on AWS

Ederson Brilhante — Tue, 12 Aug 2025 14:51:47 +0000

Introduction

GitHub Actions is the go-to CI/CD tool for many teams. But when your organization runs thousands of pipelines daily, the default setup breaks down. You hit limits on scale, security, and governance — plus skyrocketing costs.

GitHub-hosted runners are easy but expensive and don’t meet strict compliance needs. Existing self-hosted solutions like Actions Runner Controller (ARC) or Terraform EC2 modules don’t fully solve multi-tenant isolation, automation, or centralized control.

ForgeMT, built inside Cisco’s Security Business Group, fills that gap. It’s an open-source AWS-native platform that manages ephemeral runners with strong tenant isolation, full automation, and enterprise-grade governance.

This article explains why ForgeMT matters and how it works — providing a practical look at building scalable, secure GitHub Actions runner platforms.

Why Enterprise CI/CD Runners Fail at Scale

At large organizations, scaling GitHub Actions runners encounters four key bottlenecks:

Fragmented Infrastructure:
Teams independently choose their CI/CD tools: Jenkins, Travis, CircleCI, or self-hosted runners—which accelerates local delivery but creates duplicated effort, configuration drift, and fragmented monitoring. Without a unified platform, scalability, security, and reliability degrade.

Weak Tenant Isolation:
Runners run untrusted code across teams. Without strong isolation, one compromised job can leak credentials or escalate attacks across tenants. Poor audit trails slow breach detection and hinder compliance.

Scalability Limits:
Static IP pools cause IPv4 exhaustion, and manual provisioning delays runner startup. Without elastic scaling, resources are wasted or pipelines queue up, killing developer velocity.

Maintenance and Governance Overhead:
Uneven patching weakens security, infrastructure drift complicates troubleshooting, and audits become expensive and error-prone. Secure scaling demands centralized governance, consistent policy enforcement, and automation.

In short, enterprises fail to scale GitHub Actions runners without a platform that:

Centralizes multi-tenancy
Automates lifecycle management
Provides enterprise-grade observability and governance

But beware—over-centralization can kill flexibility and introduce new challenges.

Why GitHub Actions — And Why It’s Not Enough at Enterprise Scale

GitHub Actions is popular because it offers:

Deep GitHub integration: triggers on PRs, branches, and tags with no extra logins, plus automatic secret and artifact handling.
Extensible ecosystem: thousands of marketplace actions simplify workflow creation.
Flexible runners: GitHub-hosted runners for convenience, or self-hosted for control, cost savings, and compliance.
Granular security: native GitHub Apps, OIDC tokens, and fine-grained permissions enforce least privilege.
Rapid scale: pipelines at repo or org level enable smooth CI/CD growth.

However, GitHub Actions alone can’t meet enterprise-scale demands. Enterprises require:

Strong tenant isolation and centralized governance across thousands of pipelines.
A unified platform to avoid fragmented infrastructure and scaling bottlenecks.
Fine-grained identity, network controls, and compliance enforcement.
Automation for onboarding, patching, and auditing to reduce operational overhead.

Cloud providers like AWS supply identity, networking, and automation building blocks—IAM/OIDC, VPC segmentation, EC2, EKS (needed to build secure, scalable, multi-tenant CI/CD platforms).

Existing Solutions and Why They Fall Short

Actions Runner Controller (ARC) runs ephemeral Kubernetes pods as GitHub runners, scaling dynamically with declarative config and Kubernetes-native integration. But:

Kubernetes namespaces alone don’t provide strong security isolation.
No native AWS IAM/OIDC integration.
Lacks onboarding, governance, and audit automation.
Network policy management is manual, increasing operational overhead.

Terraform AWS GitHub Runner Module provisions EC2 self-hosted runners with customizable AMIs, integrating well with IaC pipelines. However:

Typically deployed per team, causing fragmentation.
No native multi-tenant isolation.
Requires manual IAM and account setup.
No onboarding or patching automation.

Commercial Runner-as-a-Service options offer simple UX, automatic scaling, and vendor-managed maintenance with SLAs, but:

High costs at scale.
Vendor lock-in risks.
Limited multi-tenant isolation.
Often don’t meet strict compliance requirements.

Where ForgeMT Fits In

ForgeMT combines the best of these approaches to deliver an enterprise-ready platform:

Orchestrates ephemeral runners seamlessly.
Uses AWS-native identity and network isolation (IAM/OIDC).
Built-in governance with full lifecycle automation.
Designed for large, security-focused organizations.

ForgeMT doesn’t reinvent ARC or EC2 modules but extends them with:

Strict multi-tenant isolation: Each team runs in a separate AWS account to contain blast radius. IAM/OIDC enforces least privilege. Calico CNI manages Kubernetes network segmentation.
Full automation: Tenant onboarding, runner patching, centralized monitoring, and drift remediation happen automatically, cutting manual toil and errors.
Centralized control plane: One dashboard securely manages all tenants with governance, audit logs, and compliance-ready traceability.
Cost optimization: Spot instances, warm pools, and autoscaling based on real-time metrics and spot prices reduce costs without sacrificing availability.
Open-source transparency: 100% open source—no vendor lock-in, no license fees, full customization freedom.

Architecture Overview

At its core, ForgeMT is a centralized control plane that orchestrates ephemeral runner provisioning and lifecycle management across multiple tenants running on both EC2 and Kubernetes.

Key Components

Terraform module for EC2 runners — provisions ephemeral EC2 runners with autoscaling, spot/on-demand, and ephemeral lifecycle.
Actions Runner Controller (ARC) — manages EKS-based runners as Kubernetes pods with tenant namespace isolation.
OpenTofu + Terragrunt — Infrastructure as Code managing tenant/account/region deployments declaratively.
IAM Trust Policies — secure runner access with ephemeral credentials via role assumption.
Splunk & Observability — centralized logs and metrics per tenant.
Teleport — secure SSH access to ephemeral runners for auditing and debugging.
EKS + Calico CNI — scalable pod networking with strong tenant segmentation and minimal IP usage.
EKS + Karpenter — demand-driven node autoscaling with spot and on-demand instances, plus warm pools.

ForgeMT Control Plane

The control plane is the platform’s brain — managing runner provisioning, lifecycle, security, scaling, and observability.

Centralized Orchestration: Decides when and where to spin up ephemeral runners (EC2 or Kubernetes pods).
Multi-Tenant Isolation: Isolates each tenant via dedicated AWS accounts or Kubernetes namespaces, IAM roles, and network policies.
Security Enforcement: Applies hardened runner configurations, automates ephemeral credential rotation, and enforces least privilege.
Scaling & Optimization: Integrates with Karpenter and EC2 autoscaling to scale runners up/down with demand and cost awareness.
Observability & Governance: Streams logs and metrics to Splunk; provides audit trails and compliance dashboards.

Runner Types and Usage

Tenant Isolation

Each ForgeMT deployment is single-tenant and region-specific. IAM roles, policies, VPCs, and services are scoped exclusively to that tenant-region pair. This hard boundary prevents cross-tenant access, simplifies compliance, and minimizes blast radius.

EC2 Runners

Ephemeral VMs booted from Forge-provided or tenant-custom AMIs.
Jobs run directly on VMs or inside containers.
IAM role assumption replaces static credentials.
Terminated after each job to avoid drift or leaks.

EKS Runners

Managed by ARC as Kubernetes pods in tenant namespaces.
Images pulled from Forge or tenant ECR repositories.
Scales dynamically for burst workloads.

Warm Pools and Limits

ForgeMT supports warm pools of pre-initialized runners to minimize cold start latency—especially beneficial for EC2 runners with slower boot times.

Per-tenant limits enforce:

Max concurrent runners
Warm pool size
Runner lifetime (auto-termination after jobs)

These controls prevent resource abuse and keep costs predictable.

Tenant Onboarding

Deploying a new tenant is straightforward and fully automated via a single declarative config file, for example:

gh_config:
  ghes_url: ''
  ghes_org: cisco-open
tenant:
  iam_roles_to_assume:
    - arn:aws:iam::123456789012:role/role_for_forge_runners
  ecr_registries:
    - 123456789012.dkr.ecr.eu-west-1.amazonaws.com
ec2_runner_specs:
  small:
    ami_name: forge-gh-runner-v*
    ami_owner: '123456789012'
    ami_kms_key_arn: ''
    max_instances: 1
    instance_types:
      - t2.small
      - t2.medium
      - t2.large
      - t3.small
      - t3.medium
      - t3.large
    pool_config: []
    volume:
      size: 200
      iops: 3000
      throughput: 125
      type: gp3
  large:
    ami_name: forge-gh-runner-v*
    ami_owner: '123456789012'
    ami_kms_key_arn: ''
    max_instances: 1
    instance_types:
      - c6i.8xlarge
      - c5.9xlarge
      - c5.12xlarge
      - c6i.12xlarge
      - c6i.16xlarge
    pool_config: []
    volume:
      size: 200
      iops: 3000
      throughput: 125
      type: gp3
arc_runner_specs:
  dind:
    runner_size:
      max_runners: 100
      min_runners: 1
    scale_set_name: dependabot
    scale_set_type: dind
    container_actions_runner: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/actions-runner:latest
    container_requests_cpu: 500m
    container_requests_memory: 1Gi
    container_limits_cpu: '1'
    container_limits_memory: 2Gi
    volume_requests_storage_type: gp2
    volume_requests_storage_size: 10Gi
  k8s:
    runner_size:
      max_runners: 100
      min_runners: 1
    scale_set_name: k8s
    scale_set_type: k8s
    container_actions_runner: 123456789012.dkr.ecr.eu-west-1.amazonaws.com/actions-runner:latest
    container_requests_cpu: 500m
    container_requests_memory: 1Gi
    container_limits_cpu: '1'
    container_limits_memory: 2Gi
    volume_requests_storage_type: gp2
    volume_requests_storage_size: 10Gi

The ForgeMT platform uses this config to:

Provision tenant-specific AWS accounts and resources.
Set IAM roles with least privilege trust policies.
Configure GitHub integration and runner specs.
Enforce tenant limits and runner types.

This automation enables zero-touch onboarding with no manual AWS or GitHub setup required by the tenant.

Extensibility

ForgeMT lets tenants customize their environments and control runner access:

Custom AMIs for EC2 runners with tenant-specific tooling.
Private ECR repositories to host container images for VMs or Kubernetes.
Tenant IAM roles with trust policies so ForgeMT runners assume them securely without static keys.
Advanced access patterns like chained role assumptions or resource-based policies for complex needs.

This lets each team tune cost, security, and performance independently without affecting core platform stability.

Security Model

ForgeMT’s foundation is strong isolation and ephemeral execution to reduce risk:

Dedicated IAM roles, namespaces, and AWS accounts per tenant.
No cross-tenant visibility or access.
Ephemeral runners destroyed immediately after job completion to prevent credential or data leakage.
Temporary credentials via IAM role assumption replace static AWS keys.
Fine-grained access control configurable by tenants for resource permissions.
Full audit trail of provisioning, execution, and shutdown logged via CloudWatch → Splunk.
Meets CIS Benchmarks and internal security policies.

Debugging in a Secure, Ephemeral World

Ephemeral runners mean persistent debugging isn’t possible by design, but ForgeMT offers:

Live debugging with Teleport: Keep runners alive temporarily via workflow tweaks to enable SSH into running jobs.
Reproducible reruns: Failed jobs can be rerun identically from GitHub UI.
Log-based troubleshooting: Access runner telemetry, syslogs, and job logs centrally without infrastructure exposure.
Kubernetes support: Same debugging mechanisms apply to EKS runners, preserving isolation and auditability.

Conclusion

ForgeMT is likely overkill for small teams. Start simple with ephemeral runners (EC2 or ARC), GitHub Actions, and Terraform automation. Only scale up when you hit real pain points. ForgeMT shines in multi-team environments where tenant isolation, governance, and platform automation are mission-critical. For solo teams, it just adds unnecessary complexity.

ForgeMT addresses the major enterprise challenges of running GitHub Actions runners at scale by delivering:

Strong multi-tenant isolation
Fully automated lifecycle management and governance
Flexible runner types with cost-aware autoscaling and warm pools
Secure, ephemeral environments that meet compliance needs
An open-source, extensible platform for customization

For organizations struggling to scale self-hosted runners securely and efficiently on AWS, ForgeMT provides a battle-tested, transparent platform that combines AWS best practices with developer-friendly automation.

Dive Into the ForgeMT Project

Ideas are cheap — execution is what counts. ForgeMT’s source code is public — check it out:

👉 https://github.com/cisco-open/forge/

⭐️ If you find it useful, don’t forget to drop a star!

🤝 Connect

Let’s connect on LinkedIn and GitHub.

Learn how ForgeMT simplifies multi-tenant GitHub Actions runners with security, scalability, and automation. Read the full case study to see how it can streamline your CI/CD pipelines:

Ederson Brilhante — Sat, 17 May 2025 10:14:18 +0000

Ederson Brilhante

May 16 '25

ForgeMT: A Scalable, Secure Multi-Tenant GitHub Runner Platform at Cisco

#terraform #devops #platformengineering #githubactions

Comments 1

14 min read

ForgeMT: A Scalable, Secure Multi-Tenant GitHub Runner Platform at Cisco

Ederson Brilhante — Fri, 16 May 2025 08:55:41 +0000

🧭 Why ForgeMT Exists

ForgeMT is a centralized platform that enables engineering teams to run GitHub Actions securely and efficiently — without building or managing their own CI infrastructure.

It provides ephemeral runners (EC2 or Kubernetes), strict tenant isolation, and full automation behind a hardened, shared control plane.

Before ForgeMT, every team in Cisco’s Security Business Group had to build and maintain their own CI setup — leading to duplicated effort, inconsistent security, slow onboarding, and rising operational overhead.

ForgeMT replaced this fragmented approach with a secure, scalable, multi-tenant platform — saving time, reducing risk, and accelerating adoption.

⚡ Fast Facts (ForgeMT Impact)

⏱️ 80+ engineering hours saved/month per team
📦 40,000+ GitHub Actions jobs/month
✅ 99.9% success rate across tenants

This post explains:

🚀 Why ForgeMT was needed
💼 What impact it had — From reliability to cost savings and security compliance.
🧱 How it works - Deep Dive into Architecture

👉 Jump to what matters most:

💼 Business Impact – For leadership and stakeholders
🏗️ Architecture – For platform engineers and DevOps
🧠 Or keep reading for full technical context and background

🚨 From Fragmented CI to Scalable, Secure Solutions: The Journey to ForgeMT

Credit: Prototype by Matthew Giassa - MASc, EIT —who championed the Philips Labs GitHub Runner module across multiple teams.

Before ForgeMT, each team used its own CI stack—Jenkins, Travis, or Concourse. While these tools met local needs, they created long-term issues: inconsistent patching, security gaps, and poor scalability.

Matthew built a promising PoC, but it was a siloed setup with manual AWS, GitHub, and Terraform steps. Rigid subnetting caused IPv4 exhaustion, and teams copy-pasting Terraform modules led to high maintenance overhead and config drift.

To address this complexity, I drove the end‑to‑end technical design and implementation of ForgeMT—a centralized, multi‑tenant GitHub Actions runner service on AWS—while coordinating with infrastructure, security, and platform stakeholders to ensure a smooth production launch.

At scale, teams were running thousands of Actions jobs across dozens of isolated environments—each with its own patch cadence, network quirks, and IAM policies. ForgeMT unifies these into a single control plane, delivering consistent security, predictable performance, and dramatically simplified operations.

For detailed business impact metrics (time saved, reliability gains, cost optimization), see the Business Impact section.

It builds on proven ephemeral EC2 and EKS/ARC runner modules, adding:

IAM/OIDC-based tenant isolation
Built-in observability (metrics, logs, dashboards)
Automation for patching, Terraform drift, repo onboarding, and global Actions locks

By consolidating infrastructure into a hardened control plane, ForgeMT ensured that security and compliance were at the forefront while enabling rapid onboarding, eliminating manual patching, and solving IPv4 exhaustion. This was achieved by scaling pod-based runners via EKS + Calico CNI, with a strong focus on tenant isolation, IAM roles, and security groups (SG) to control access. The hardened control plane preserved the security and flexibility of the original prototype, delivering a secure, compliant, and scalable platform.

📊 Business Impact

ForgeMT has not only met the demands of various teams but also helped scale securely under Cisco’s guidance, optimizing cloud spend and increasing reliability across all stakeholders:

Dramatic time savings (80 + hours/month per team): By automating every aspect of runner lifecycle—OS patching, Terraform module updates, ephemeral provisioning, and even repository registration—teams were freed from manual CI maintenance and could refocus on shipping features.
Optimized cloud spend: Spot and On-demand Instances, right‑sized instance selection per job type, and EKS + Calico’s IP‑efficient networking cut infrastructure costs without slowing builds. ForgeMT also supports warm instance pools for high-frequency jobs, avoiding cold starts when speed is critical—striking a smart balance between performance and cost.
Rock‑solid reliability (99.9% success over 40K+ jobs/month): Centralizing infrastructure eliminated snowflake environments and drift, reducing job failures caused by misconfiguration or stale runners to near zero.
Enterprise‑grade security & compliance: IAM/OIDC per‑tenant isolation, CIS‑benchmarked AMIs, and end‑to‑end logging into Splunk ensured every action was auditable, vault‑grade credentials were never exposed, and internal audits passed with zero findings.
True multi‑tenancy at scale: Teams retain autonomy over AMIs, ECRs, and workflow definitions while ForgeMT transparently handles networking, isolation, and autoscaling—supporting dozens of teams without additional IP consumption or operational overhead.
AWS account isolation per tenant: Each tenant can have one or more individual AWS accounts, with full control over their own network setup. This includes the flexibility to configure internal or public subnets within their AWS accounts, ensuring strong security boundaries and independent resource management without ForgeMT managing their network.

Together, these outcomes turned a fractured, high‑toil CI landscape into a self‑service platform that scales securely, reduces costs, and accelerates delivery.

⚙️ ForgeMT Architecture Overview

📦 Core Components & Technical Foundations

These results are enabled by the following technical components:

Terraform module for EC2 runners: Utilized as a Terraform module to provision ephemeral EC2-based GitHub Actions runners, supporting auto-scaling and cost optimization by using AWS spot and on-demand instances. This setup ensures that runners are created on-demand and terminated after use, aligning with the ephemeral nature of ForgeMT's infrastructure.
ARC (Actions Runner Controller): Employed to manage EKS-based GitHub Actions runners, enabling containerized, isolated job execution via Kubernetes. This approach leverages Kubernetes' orchestration capabilities for efficient scaling and management of CI/CD workloads.
OpenTofu + Terragrunt: Implemented for Infrastructure as Code (IaC), ensuring region-, account-, and tenant-specific infrastructure deployments with DRY (Don't Repeat Yourself) principles. This methodology facilitates consistent and repeatable infrastructure provisioning across multiple environments.
IAM Trust Policies: Adopted to secure runner access using short-lived credentials via IAM roles and trust relationships, eliminating the need for static credentials and enhancing security.
Splunk Cloud & O11y(Observability): Integrated for centralized logging and metrics aggregation, providing real-time observability across ForgeMT components. This setup enables detailed telemetry, including per-tenant dashboards for monitoring resource usage and optimization insights.
Teleport: Utilized to provide secure, auditable SSH access to EC2 runners and Kubernetes pods, enhancing compliance, access control, and auditing capabilities.
EKS + Calico CNI: Leveraged to scale pod provisioning without consuming additional VPC IPs, utilizing Calico's efficient networking. This setup ensures tenant isolation and optimizes network resource usage within limited VPC subnets.
EKS + Karpenter: Enables dynamic, demand-driven autoscaling of Kubernetes worker nodes. Automatically provisions the most suitable and cost-effective EC2 instance types based on real-time pod requirements. Supports spot and on-demand capacity, prioritizing efficiency and performance. Warm pools can be configured to reduce cold start latency while maintaining cost control—ideal for high-churn CI/CD workloads.

These technologies form the backbone of ForgeMT, enabling its robust performance and scalability.

🧠 ForgeMT Control Plane (Managed by Forge Team)

The ForgeMT control plane hosts shared infrastructure and reusable IaC modules:

ForgeMT GitHub App: Installed on tenant repositories to listen for GitHub workflow events and dynamically register ephemeral runners.
ForgeMT AMIs & Forge ECR: Default base images for runners (VMs and containers).
Terraform Modules: Each tenant-region pair deploys an isolated ForgeMT instance.
API Gateway + Lambda: Processes GitHub webhook jobs to trigger runner provisioning.
Centralized Logging: Runner logs are forwarded to CloudWatch, then into Splunk Cloud Platform.
Centralized Observability: All AWS metrics are sent to Splunk O11y Cloud
Teleport: Secure, role-based SSH access to VM runners (if needed), with session logging.

🏗️ Tenant Isolation

Each ForgeMT deployment is dedicated to a single tenant, ensuring full isolation within a specific AWS region. This approach guarantees that IAM roles, policies, services, and AWS resources are scoped uniquely for each tenant-region pair, enforcing strict security, compliance, and minimizing the blast radius.

💻 Runner Types

🧱 AWS EC2-Based Runners (VM and Metal)

Ephemeral Runner Provisioning: EC2 runners are provisioned using Forge-provided AMIs or tenant-specific custom AMIs. These instances are pre-configured with the necessary tools to execute CI/CD jobs.
Workload Execution: Jobs can be executed directly on the EC2 instance or via containers, using container: blocks in GitHub workflows.
Security: Authentication to tenant AWS resources is handled through IAM roles and trust policies, eliminating the need for static credentials and ensuring dynamic, secure access control.
Ephemeral Nature: Once a job is completed, the EC2 instance is terminated, maintaining a completely stateless environment.

☸️ EKS-Based Runners (Kubernetes)

Kubernetes-Orchestrated Actions: Using the Actions Runner Controller (ARC), EKS runners are provisioned as pods within an Amazon EKS cluster.
Resource Isolation: Each tenant is assigned a dedicated namespace, service account, and IAM role, ensuring strict isolation of resources and permissions.
Container Images: Runners can pull container images from either the Forge ECR or the tenant’s own ECR, depending on the configuration.
Scalability: EKS is ideal for high-scale operations, leveraging Kubernetes' orchestration capabilities to manage the lifecycle of runners efficiently.

🔁 Warm Pool

Reducing Startup Latency: An optional warm pool can be configured for both EC2 and EKS runners, pre-initializing instances or pods to reduce waiting times during high demand.
Importance for EKS: For EKS runners, the need for warm pools is significantly reduced, as Kubernetes already provides rapid scaling and efficient pod initialization.
Usage in EC2: The warm pool helps minimize the initialization time for EC2 instances, resulting in faster job execution times for critical tasks.

💻 Examples of Runner Types in ForgeMT

ForgeMT offers flexibility for tenants to configure multiple runner types simultaneously, adapting to their workload needs. Each tenant can define as many runners as needed, with a parallelism limit set per tenant and runner type. Here are some typical runner examples and their use cases:

🧱 EC2 Runners

Small: Lightweight instances for tasks with minimal resource usage, such as quick tests or linting.
Standard: Instances for balanced workloads, ideal for code compilation or integration tests.
Large: High-performance instances for tasks requiring more processing power, such as complex builds or load tests.
Bare Metal: Bare-metal instances for applications that need full control over the hardware, such as simulations or intensive processing tasks.

☸️ Kubernetes Runners

Dependabot: Used for automated dependency update jobs.
Light (k8s): Runners for simple tasks that don't require Docker, like linting or unit test execution.
Docker-in-Docker (DinD): Used for jobs that require Docker inside Kubernetes, such as image building or integration tests involving containers.

🔄 Configurable Parallelism per Tenant and Runner Type

Each tenant can configure their own set of runners and use different EC2 instance types or Kubernetes pods simultaneously.
The parallelism limit can be configured per runner type and tenant, ensuring that running multiple jobs does not overload resources.
This allows each team to run jobs in parallel based on their needs without impacting the performance of other tenants or jobs.

Considerations

Choosing the Right Runner: Depending on workload complexity and job requirements, you may choose EC2 or EKS runners. EKS is generally preferred for lightweight, scalable workloads, while EC2 may be necessary for jobs with specific hardware or memory requirements.

⚙️ GitHub Integration

GitHub events trigger ForgeMT through a webhook via API Gateway, dynamically registering runners into the appropriate GitHub Runner Groups associated with the tenant. The runner lifecycle is designed to be ephemeral: runners are registered just-in-time for job execution and are destroyed once the job is completed. When a new repository is installed, it is automatically registered with the correct GitHub Runner Group, ensuring seamless integration with the right tenant's runners.

🔌 Extensibility

Each tenant account can optionally manage the following resources:

Tenant AMIs (for AWS EC2 runners): Custom-built images with pre-installed tooling tailored to the tenant's specific requirements.
Tenant ECR: Houses custom container images used for VM-based container jobs, GitHub composite actions, or full pod images in EKS.
Tenant IAM Role: Configured with trust relationships to allow ForgeMT runners to securely assume roles without the need for AWS access keys.
ForgeMT offers flexibility for teams to customize their runners according to their specific needs. If a tenant requires a custom Amazon Machine Image (AMI) or container image, it is their responsibility to build and maintain it. We provide a base image to get them started, but the final configuration is under their control. Once the custom image is ready, it can be shared with our accounts and integrated into the ForgeMT platform, enabling the team to meet their unique requirements.

🔄 Optional Configurations

Tenants can choose to configure the following based on their specific needs:

Accessing AWS Resources via Runners: To enable runners to interact with AWS services within the tenant's account, an IAM role must be established with a trust relationship permitting ForgeMT to assume it.
Pulling Images from Tenant ECR: If runners need to pull images from the tenant's ECR—be it for container jobs, composite actions, or Kubernetes pods—the tenant must configure appropriate repository policies and IAM permissions to allow these operations.
Accessing Additional Tenant Resources: For runners to access other AWS resources within the tenant's account, the IAM role assumed by ForgeMT must have policies granting the necessary permissions. This might involve setting up a chain of role assumptions or defining specific resource-based policies.

📊 Observability: Splunk Cloud & O11y

ForgeMT delivers full-stack observability with centralized logging and per-tenant metrics:

Centralized logging: All relevant logs — syslog, AWS EC2 user data, GitHub runner job logs, worker logs, and agent logs — are sent to CloudWatch Logs and forwarded to Splunk Cloud for full visibility and auditability.
Metrics via Splunk O11y: Captures detailed telemetry.
Per-tenant dashboards: Each team gets dedicated dashboards showing cost breakdowns, resource usage, and optimization insights (e.g., high-memory job detection).

🔐 Security Model

Strong tenant isolation: Every tenant has its own IAM roles, namespaces, and resources.
IAM Role Assumption: Eliminates use of long-lived AWS credentials.
No cross-tenant visibility: Runners cannot access other tenant workloads or secrets.
Fine-grained access control: Each tenant defines what their runners can access by configuring the IAM role being assumed—this can include direct resource access or chained role assumptions for more advanced patterns.
🔒 Ephemeral Isolation: ForgeMT runners are automatically destroyed after every job — success or failure. This guarantees a clean slate every time, eliminates environment drift, blocks credential persistence, and prevents resource leaks by default.

🛡️ Compliance & Observability

ForgeMT ensures strict compliance and security throughout the lifecycle of its ephemeral runners, from provisioning to execution and shutdown.

Full Audit Trail: Every runner lifecycle event — including provisioning, execution, and shutdown — is logged, ensuring complete visibility and traceability for compliance audits. This audit trail is vital for maintaining transparency in high-security environments.
CloudWatch → Splunk Integration: Logs from the runners are forwarded from CloudWatch to Splunk, enabling teams to perform real-time queries on logs. This integration supports compliance audits by providing detailed, queryable logs that can be easily reviewed and accessed for regulatory requirements.
IAM Integration: By using IAM (Identity and Access Management), ForgeMT eliminates the use of hardcoded credentials or AWS long-term access keys. This significantly reduces the risk of unauthorized access and enhances security by enforcing role-based access and temporary credentials that follow the principle of least privilege.
Security Standards Compliance: ForgeMT meets internal security standards, which are aligned with industry best practices such as CIS Benchmarks. This ensures that the platform adheres to rigorous security controls and provides a secure environment for multi-tenant workloads.

🔍 Debugging Securely and Effectively

ForgeMT offers teams the option to choose between EC2 Spot Instances and On-Demand Instances, allowing for flexibility in cost optimization. While Spot Instances can provide significant cost savings, they come with the inherent risk that AWS may reclaim the instance at any time. Teams are responsible for evaluating this risk and determining whether to use Spot or On-Demand Instances based on the criticality of their workloads.

Given ForgeMT's design of ephemeral runners, which are terminated immediately after each job to prevent state persistence and credential leakage, debugging presents unique challenges. However, the platform offers robust solutions to address these challenges.

For real-time debugging, developers can access running jobs via Teleport. By including a sleep step in the workflow or using a custom wrapper, the runner can be kept alive temporarily. This allows for manual inspection and troubleshooting while the job is still running.

Additionally, even without live access, ForgeMT maintains comprehensive observability. Teams can rely on syslogs, GitHub Actions job logs, and runner-level telemetry to understand job behavior. Every job runs in a fully reproducible environment, meaning developers can simply rerun failed jobs through the GitHub UI, replicating the exact conditions without side effects while maintaining full auditability.

For Kubernetes-based runners, the same debugging approach applies: Teleport can be used for live access to running jobs. The integration with Kubernetes allows teams to extend the same debugging capabilities while leveraging the scalability and flexibility of the containerized environment.

🚀 ForgeMT: Powering Tenants with Flexibility and Control

💥 Ephemeral by design — Runners are created per job and disappear afterward. No drift. No patching. No residual garbage.
🛠️ Infra-as-Code from top to bottom — Fully automated. Declarative. Version-controlled. No snowflakes.
🔐 Strong isolation baked in — IAM, OIDC, and security group segmentation per tenant. No cross-tenant blast radius.
📦 Run anything, per tenant — EC2 or EKS. k8s, dind, or metal. Each tenant defines their own mix.
🚦 Control usage at scale — Enforce parallelism limits per tenant/type. No surprises. No abuse.
🕹️ Custom policies, zero effort — Tenants define autoscaling, labels, and configurations via GitHub — no AWS skills required.
🧘 No infra for tenants to manage — No patching, no VPCs, no accounts. Just push code.
🕵️ Observability without ownership — Logs, metrics, and traces exposed per tenant. No nodes to babysit.
⚡ Fast time-to-first-run — Cold starts optimized. Most runners boot in <20s, even for large jobs.
🌎 Network-aware provisioning — Runners automatically deploy into the correct subnet, zone, or region.
📊 Usage-aware scaling — Instance types are selected based on cost/performance tradeoffs — no more overprovisioning by default.
🧩 GitHub-native workflows — No toolchain rewrites required. Just drop in the runs-on labels and go.
🚫 No global queues — Each tenant is scoped, isolated, and throttled independently.

🛠️ Implementation & Adoption

It took about 2 months to evolve from a single-tenant, EC2-only setup into a fully multi-tenant platform. Highlights:

🔹 *Kubernetes support *— with Calico CNI + Karpenter
🔹 Tenant isolation by design
🔹 Per-tenant automation & base images
🔹 EKS pod identity for secure access
🔹 Integrated with Teleport, Splunk, and full observability
🔹 Custom dashboards with enriched telemetry

🚀 Frictionless Adoption

Onboarding was dead simple.

For most tenants, switching to ForgeMT meant updating just the runs-on label in their GitHub Actions workflows — ⚡ No rewrites. No migrations. No downtime.

For teams that required deeper isolation, assuming their own IAM role was just as straightforward:

- name: Configure AWS Credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::<tenant-account>:role/<role-name>
    aws-region: <aws-region>
    role-duration-seconds: 900

- name: Example
  run: aws cloudformation list-stacks

💡 This approach made adoption fast, safe, and low-friction — even for teams skeptical of platform changes.

🚫 Overkill Warning: When ForgeMT Is Too Much

If you're a small team, ForgeMT might be overkill. Start with the basics: ephemeral runners (EC2 or ARC), GitHub Actions, and Terraform automation. Scale up only when you hit real pain. ForgeMT shines in multi-team setups where governance, tenant isolation, and platform automation matter. For solo teams, it may just add complexity you don’t need.

🔭 What’s Next

I’m currently focused on:

Cost-aware scheduling — Prioritizing jobs based on real-time pricing and instance efficiency, optimizing for performance while reducing costs.
Dynamic autoscaling — Moving from static warm pool rules to a more responsive, metrics-driven approach that adapts to the bursty nature of GitHub Actions workloads.
Deeper observability — Integrating GitHub metrics for actionable insights that drive optimized runner performance.
AI-driven scaling optimization — Leveraging historical data to predict workload demands, optimize resource allocation, and automate scaling decisions based on both performance and cost metrics.

If you’re tackling similar problems — or looking to adopt, extend, or contribute to ForgeMT — let’s talk. I’m always open to collaborating with engineers building serious DevSecOps infrastructure.

🧪 Dive Into the ForgeMT Project

Ideas are cheap — execution is everything. The ForgeMT source code is now publicly available — check it out:

👉 https://github.com/cisco-open/forge/

⭐️ Don’t forget to give it a star ;)!

✍️ In Short

ForgeMT emerged from real-world CI pain at enterprise scale. What began as a prototype to fix local inefficiencies has grown into a secure, multi-tenant, production-grade runner platform. I’m sharing this so others can skip the trial-and-error and build smarter from the start.

🤝 Connect

Let’s connect on LinkedIn and GitHub.

Always happy to trade notes with like-minded builders.

This article was originally published on LinkedIn.

Decoding the Myth of 'Junior' in DevOps and SRE: Navigating Challenges and Cultivating Expertise

Ederson Brilhante — Tue, 16 Jan 2024 15:38:01 +0000

In my view, assigning roles such as 'Junior DevOps' and 'Junior SRE (Site Reliability Engineer)' seems impractical, reminiscent of labeling someone an 'Entry-Level Software Architect.'

Navigating the intricate landscape

Navigating the intricate landscape of DevOps and SRE demands proficiency in coding, networking, cloud technologies, security, and system administration. Envisioning someone with limited experience adeptly maneuvering through this multifaceted skill set poses a significant challenge.

The Software Architect analogy

Similarly, giving the title "Software Architect" to beginners doesn't align with the intricate demands of the role. Crafting sophisticated software solutions requires years of practical experience, involving intricate system design and understanding. Expecting a junior engineer to architect and implement a secure, scalable microservices architecture without in-depth knowledge and experience in the design principles of distributed systems is unrealistic.

Quantity vs. Experience fallacy

Furthermore, the belief that numerous junior roles collectively can achieve the same level of effectiveness as a seasoned professional echoes the fallacy of favoring quantity over experience. While each junior role contributes to the team's growth, the efficiency and strategic thinking of an experienced architect often outpace the combined efforts of multiple entry-level professionals.

Pressure on companies

In addition, the pressure on companies to leverage the benefits of DevOps and SRE roles within their organization often stems from the growing need for seamless integration between development and operations. Individuals in these positions are expected to possess a profound understanding of both coding and operations, creating a unique blend of skills. Unfortunately, finding professionals who embody this multidisciplinary expertise is a formidable challenge. Those who can seamlessly bridge the gap between traditional sysadmins and developers are not only rare but also come at a premium, given the scarcity of individuals with such comprehensive skills in the overall job market.

Scarcity leading to desperation

This scarcity sometimes leads companies to consider entry-level candidates, hoping to quickly train them to fill the void. However, the complex nature of the disciplines touched upon by DevOps and SRE roles means that becoming proficient in each area takes years of hands-on experience. The high demand and limited supply of individuals with these multifaceted skills contribute to the desperation companies feel in recruiting for these roles.

Acknowledging the shortage

Acknowledging this shortage is crucial, especially as it extends beyond DevOps and SRE roles to other senior positions. Over the past two decades, the industry has witnessed a trend of companies poaching professionals from one another rather than investing in training new talents. This cycle has created a snowball effect, further exacerbating the shortage of skilled individuals.

Solution: Attracting seasoned developers

A potential solution lies in attracting seasoned developers with a penchant for infrastructure and operations to transition into roles in DevOps and SRE. These individuals often bring a wealth of experience, having naturally acquired knowledge in areas beyond coding, such as security, infrastructure, databases, and operations. Their diverse skill set aligns with the demands of contemporary senior developers who are expected to possess expertise beyond language-specific coding skills. By encouraging such transitions, companies can tap into a pool of experienced professionals and mitigate the challenges associated with the scarcity of multidisciplinary talent in the market.

Recommended pathway for aspiring professionals

For aspiring professionals entering the tech industry, a recommended pathway involves starting as a developer before venturing into the multifaceted realms of DevOps and SRE. Beginning as a developer allows individuals to hone their coding skills and gain a solid foundation in software engineering principles. As they accumulate experience and familiarity with the development lifecycle, they can then gradually navigate towards operations, infrastructure, and other related disciplines. This gradual journey not only provides a comprehensive understanding of the intricacies of both coding and operations but also allows individuals to develop a deeper appreciation for the challenges addressed by DevOps and SRE roles. This approach acknowledges the value of hands-on experience and ensures that individuals entering these dynamic fields are well-equipped to contribute meaningfully to the integration of development and operations within an organization.

Combining Packer, QEMU, Ubuntu Cloud Images, and Ansible

Ederson Brilhante — Wed, 14 Jun 2023 20:50:24 +0000

Hello everyone! I want to share a current use case at my company where I have the opportunity to work with Packer, QEMU, Ansible and Ubuntu Cloud Images leveraging the concept of Infrastructure as Code (IaC).

Infrastructure as Code (IaC) is a software engineering practice that enables the management and provisioning of infrastructure resources through code. Instead of manually configuring servers and infrastructure components, IaC allows you to define your desired infrastructure state using declarative or imperative code. It brings automation, version control, and consistency to infrastructure management.

In our case, we utilize Packer, which is a powerful tool falling under the umbrella of IaC. Packer enables the creation of identical machine images for multiple platforms, such as virtual machines, containers, or cloud instances. With Packer, we define the configuration of our desired machine image, including the operating system, software stack, and customizations, all through code. Packer then automates the process of building these machine images, ensuring consistency and reproducibility.

To further enhance our image-building process, we integrate Ansible as the provisioner for Packer. Ansible is an open-source automation tool that enables the configuration and management of systems through simple, human-readable YAML files. With Ansible, we define the desired state of our machine image, including the installation of packages, configuration files, and any other necessary setup steps. Ansible seamlessly integrates with Packer, allowing us to provision our machine image with ease.

In our deployments, we rely on Ubuntu images to meet our diverse cloud computing needs. Ubuntu offers three types of images: live, server, and cloud. Live images provide a fully functional Ubuntu desktop environment that can be run directly from a USB drive or DVD without the need for installation, while server images are optimized for server deployments. However, for our specific use case, we have different requirements depending on the deployment environment. For our deployments in public cloud environments, we leverage the official Ubuntu images provided by the cloud provider, which are tailored and certified for their specific platform. Similarly, in our private on-premises cloud, we utilize the cloud version of Ubuntu images. These cloud images are specifically designed and pre-configured for cloud computing platforms, offering optimized performance and scalability. They enable us to efficiently deploy and manage Ubuntu instances in both our public and private cloud environments.

Now, let's delve into our challenge. The process of building VM images for the public cloud involves the use of appropriate plugins for Packer. However, when it comes to our on-premises cloud, we encountered an obstacle. Our existing process relied on a deprecated plugin relying on QEMU as its underlying technology. QEMU, an open-source virtualization tool, empowers us to operate and manage virtual machines in various formats, including qcow2. To overcome this hurdle, our aim was to leverage QEMU using an official and updated plugin for Packer. This integration would seamlessly incorporate QEMU into our image-building process, delivering enhanced efficiency and reliability.

While I had prior experience with Packer, my familiarity with QEMU was limited, especially when it came to using Packer with QEMU. To address this knowledge gap, I referred to the official documentation of Packer. However, I encountered a challenge: the documentation provided an example using a server version of CentOS, which wasn't suitable for my requirements. I needed a cloud version of Ubuntu, which does not come with default user and password credentials. To overcome this hurdle, I created a seed image that included user-data and meta-data. This seed image allows us to "emulate" the cloud-init functionality. By combining this seed image with the Ubuntu image, Packer can establish an SSH connection to the virtual machine successfully.

In the seed image, we create a user with the necessary credentials for the initial build process. It's important to note that the user created in the seed image is only intended for the build phase and is not present in the final image. This approach ensures that the final image does not contain any unnecessary or insecure credentials, maintaining a clean and secure environment.

Here's the code to generate the seed image for the packer_qemu.seed.pkr.hcl script:

source "file" "user_data" {
  content = <<EOF
#cloud-config
ssh_pwauth: True
users:
  - name: user
    plain_text_passwd: packer
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    lock_passwd: false
EOF
  target  = "user-data"
}

source "file" "meta_data" {
  content = <<EOF
{"instance-id":"packer-worker.tenant-local","local-hostname":"packer-worker"}
EOF
  target  = "meta-data"
}

build {
  sources = ["sources.file.user_data", "sources.file.meta_data"]

  provisioner "shell-local" {
    inline = ["genisoimage -output cidata.iso -input-charset utf-8 -volid cidata -joliet -r user-data meta-data"]
  }
}

In the provided code, the genisoimage command-line tool plays a crucial role in generating the necessary configuration files for cloud-init. Specifically, it is used to create the cidata.iso file, which encapsulates the user_data and meta_data files. These files contain important cloud-init configuration data, such as user credentials and metadata information for the instance.
By utilizing genisoimage, we can create a bootable ISO image that incorporates the required configuration data. This ISO image is then seamlessly integrated into the image-building process by Packer.
To gain a better understanding of the genisoimage command-line options and functionality, you can refer to the official documentation at Genisoimage Documentation. The documentation provides detailed explanations and examples to help you effectively utilize genisoimage in your image-building workflow.

Here's the code for the packer_qemu.qcow2.pkr.hcl script that uses the seed image and cloud image to build a new image and then runs an Ansible playbook to configure the new image:

packer {
  required_plugins {
    vagrant = {
      version = "1.0.9"
      source  = "github.com/hashicorp/qemu"
    }
    ansible = {
      version = "1.0.4"
      source  = "github.com/hashicorp/ansible"
    }
  }
}

source "qemu" "ubuntu" {
  format           = "qcow2"
  disk_image       = true
  disk_size        = "10G"
  headless         = true
  iso_checksum     = "file:https://cloud-images.ubuntu.com/focal/current/SHA256SUMS"
  iso_url          = "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
  qemuargs         = [["-m", "12G"], ["-smp", "8"], ["-cdrom", "cidata.iso"], ["-serial", "mon:stdio"]]
  shutdown_command = "echo 'packer' | sudo -S shutdown -P now"
  ssh_password     = "packer"
  ssh_username     = "user"
  vm_name          = "build.qcow2"
  output_directory = "output"
}

build {
  sources = ["source.qemu.ubuntu"]

  provisioner "ansible" {
    playbook_file = "ansible/qemu.yml"
  }
}

This code snippet showcases the Packer configuration language to orchestrate the build process. It begins with the packer block, which outlines the essential plugins required for Packer, including qemu and ansible. The source block focuses on configuring the QEMU source, encompassing various settings such as the format, disk size, ISO checksum and URL, QEMU arguments, SSH credentials, and more. Within the build block, the QEMU source is designated for the build process.
Additionally, the provisioner section incorporates the ansible provisioner, specifying the Ansible playbook (ansible/qemu.yml) to execute for further customization of the newly created image. For a comprehensive understanding of the packer plugin arguments pertaining to qemu, you can refer to the official documentation at Packer QEMU Plugin. The documentation offers detailed insights into the various plugin options and configurations available for QEMU integration within Packer.

By combining Packer, QEMU, Ubuntu Cloud Images, and Ansible, we are able to automate the process of building consistent and reproducible machine images for our on-premises Data Center. This streamlined approach saves time, ensures consistency across our environments, and maintains a secure image without unnecessary credentials.

I hope sharing our experience and providing these code snippets will help the community facing similar challenges in the future. Let's continue building and automating together! If you have any questions or suggestions, feel free to reach out.

Building labs using component-based architecture with Terraform and Ansible

Ederson Brilhante — Thu, 14 Apr 2022 15:48:35 +0000

Currently, I am a Site Reliability Engineer(SRE) in the observability team at Splunk. But when I worked in this solution I was part of the GDI(Get Data In) organisation at Splunk.

Now, let's talk about the problem.
Part of the engineer's job in GDI is building add-ons to Splunk. Add-ons, in a nutshell, are plugins to connect third party data sources to Splunk platform.

Every time we need to work on a new add-on version of a specific third party, we need to set up 2 labs, 1 for development purposes, and the other with QA specifications.

The GDI organisation owns many add-ons, so we use a strategy to make rotations in which team and who in the team will work in a new version every time.
This is good to spread the knowledge, but we had problems keeping reliable and consistent labs across the dev cycle and the teams.

A big fraction of people's time was manual work to set up the labs(manual configuration or writing new bash/power-shell scripts). Along with a lot of time expended in the development process, the manual work creates a great deal of headache for the developers

The teams agreed it needed some automation, in order to reduce the pain to create labs and avoid duplication or rework.

We came up with the idea of using infra as code(IaC). Which was nothing so special that other companies weren't already doing.

Because the teams are small, and they are focused on the development of add-ons, we need an approach where the teams could have customised labs, but not necessary to write IaC scripts.

Based on the Design Principles of react components, we came up with an idea to create components that can be reused and plugged in other components. And each component would be a Terraform Module, an Ansible Playbook or an Ansible Role.

For a better elucidation, let's use this example - Build a lab with 3 different environments:

Environment A will have 4 windows instances:
- 1 Windows Server 2016 as Domain Controller.
- 3 Windows Servers as members server:
  - 1 Windows Server 2016 with Windows Event Collector:
  - Splunk Universal Forward
  - Collecting only Sysmon events from nodes
  - 1 Windows Server 2016 with Windows Event Forwarding
  - 1 Windows Server 2019 with Windows Event Forwarding
Environment B will have 7 windows instances:
- 1 Windows Server 2016 as Domain Controller.
- 6 Windows Servers as members server:
  - 1 Windows Server 2016 with Windows Event Collector (WEC A):
  - Splunk Universal Forward
  - Collecting only Application events from nodes
  - 1 Windows Server 2016 with Windows Event Forwarding, sending logs to WEC A
  - 1 Windows Server 2019 with Windows Event Forwarding, sending logs to WEC A
  - 1 Windows Server 2019 with Windows Event Collector (WEC B):
  - Splunk Universal Forward
  - Collecting only Security events from nodes
  - 1 Windows Server 2016 with Windows Event Forwarding, sending logs to WEC B
  - 1 Windows Server 2019 with Windows Event Forwarding, sending logs to WEC B
Environment C will have 3 windows instances:
- 1 Windows Server 2016 as Domain Controller with Windows Event Collector :
  - Splunk Universal Forward
  - Collecting only Security and Sysmon events from nodes
- 2 Windows Servers as Members Server:
  - 1 Windows Server 2016 with Windows Event Forwarding
  - 1 Windows Server 2019 with Windows Event Forwarding

Normally, Using terraform modules and Ansible Playbooks we could reproduce these environments.
We would need to create specific playbooks and terraform configs for each environment.
And here comes the problem. Spending time coding permutations in some similar configurations.

To avoid that, our approach with component based architecture, we only have to write a single config file describing which modules these labs need to run without touching any Terraform Script or Ansible Playbook.

Architecture

The solution we made is compatible with many kind of labs configurations deployed in AWS.

Terraform scripts are used to deploy the infrastructure, spinning up EC2 instances and other AWS resources. And to provision softwares and system configuration inside of each EC2 instance, terraform calls proper ansible playbooks.

Playbooks are a group of roles. A role represents an implementation of specifics configuration in an independent way.

Take role windows_splunk_universal_forward as example. This role downloads, installs and configures a splunk universal forward instance in Windows. This role is coded to be used any windows version.

Repo Structure:

   ├── ansible
   │   ├── all_roles
   │   │   └── distros
   │   │       ├── linux
   │   │       │   └── roles
   │   │       │       └── <new-linux-role>
   │   │       └── windows
   │   │           └── roles
   │   │               └── <new-windows-role>
   │   └── playbooks
   │       └── <new-playbook>
   └── terraform
       └── modules
           ├── distros
           │   └── <new-distro-type>
           └── environments
               └── <new-environment-type>

Terraform

Terraform is an open-source infrastructure as code software tool that provides a consistent CLI workflow to manage hundreds of cloud services. Terraform codifies cloud APIs into declarative configuration files.

For more info check on official documentation.

Terraform Structure:

   terraform/
   ├── modules
   │   ├── constants
   │   ├── core
   │   ├── distros
   │   │   └── <distro-type>
   │   └── environments
   │       └── <environment-type>
   └── wire

What is an environment?

A environment is a pre-defined kind of relations between nodes.
Each module environment is found in path terraform/modules/environments. And uses the modules in terraform/modules/distros to build the proper relations.

For elucidation, take this case as an example:

1 Windows Domain Controller.
X number of Member Servers.

We have this hierarchy, because we need create first the DC and so give some data to member servers, such as IP:

```
# file: terraform/modules/environments/linux-standalone/main.tf

module "windows-domain-controller" {
    source = "../../distros/windows-server"
    ...
}

module "windows-server-member" {
    source = "../../distros/windows-server"
    ...
    windows_domain_controller = module.windows-domain-controller
    ...
}
```

What is a distro?

A distro is a pre-defined kind of AMI with specific kind of setup and/or provisioning.
Each module distro is found in path terraform/modules/distros. And have a proper ansible playbook to execute the provisioning.

For elucidation, take these cases as examples:

Linux
Windows
Splunk
Free BSD

Terraform example:

```

locals {
...
provisioning_command     = "ansible-playbook -i $PUBLIC_IP /opt/automation/tools/ansible/playbooks/windows.yml --extra-vars='${local.extra_vars}'"
}

...

resource "aws_instance" "windows_server" {
...
}

resource "null_resource" "ansible" {

triggers = {
    command = replace(local.provisioning_command, "$PUBLIC_IP", "'${aws_instance.windows_server.public_ip},'")
}

provisioner "local-exec" {
    command = replace(local.provisioning_command, "$PUBLIC_IP", "'${aws_instance.windows_server.public_ip},'")
}
}
```

Ansible

Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows.

For more info check on official documentation.

Ansible Structure:

```
ansible/
├── all_roles
│   ├── distros
│   │   └── <distro-type>
│   │       └── roles
│   │           └── <distro-role>
└── playbooks
```

What is a distro type?

A distro type is folder that centralize all ansible roles that can be used executed in a specific.

Take windows as example: ansible/all_roles/distros/windows. This folder centralize all ansible roles that can be used executed in a windows machines.

What is a distro role?

A distro role is a group of ansible tasks, that implements related configurations that represents a functionality.

For elucidation, take the list of tasks from splunk UF role:
- Downloads Splunk UF
- Installs the download file
- Sets default configuration
- Starts Splunk UF

Explaining the config file

Here you can find a complete config example:

config = {
  "myenv01" = {
    "type" = "windows_standalone"
    "nodes" = {
      "myvm01" = {
        "type" = "windows"
        "enabled_roles" = {
          "windows_funcionality01" = true
          "windows_funcionality02" = true
        }
        "os" = {
          "size"    = "t2.medium"
          "distro"  = "windows"
          "type"    = "windows"
          "version" = "2016"
        }
      }
    }
  }
  "myvm02" = {
    "type" = "linux_standalone"
    "nodes" = {
      "mylinux01" = {
        "type" = "linux"
        "enabled_roles" = {
          "linux_funcionality01" = true
          "linux_funcionality02" = true
        }
        "os" = {
          "size"    = "t2.medium"
          "distro"  = "ubuntu"
          "type"    = "linux"
          "version" = "20"
        }
      }
    }
  }
}

Under the hood this configuration will be translated to create 2 EC2 instances in AWS, and each instance will run playbooks with specific roles.

Block explanation:

Each block myenv0x represents how the environment will be deployed. The type represents which predefined environment will be used.
Each block myvm0x represents a VM that will be created. The type represents which predefined distro will be used.
The block os has 4 properties that will create a proper EC2 instance:
- The AWS type instance
- Type of Distro (windows, linux, etc)
- OS Distro(ubuntu, debian, suse, windows, etc)
- Version of the OS Distro

With this info the terraform will know which AWS AMI to use to spin up in the EC2 instance

The block enabled_roles represents a list of Ansible Roles to execute in each instance

For more details about the code and implementation, check the code demo, fully functional.

A serverless full-stack application using only git, google drive, and public ci/cd runners

Ederson Brilhante — Fri, 16 Apr 2021 13:40:49 +0000

TL;DR; How I built the Vilicus Service, a serverless full-stack application with backend workers and database only using git and ci/cd runners.

What is Vilicus?

Vilicus is an open-source tool that orchestrates security scans of container images(Docker/OCI) and centralizes all results into a database for further analysis and metrics.

Vilicus provides many alternatives to use it:

Own Installation;
GitHub Action in your GitHub workflows;
Template CI in your GitLab CI/CD pipelines;
Free Online Service;

This article explains how it was possible to build the Free Online Service without using a traditional deployment.

Architecture

The Frontend is hosted in GitHub Pages. This frontend is a landing page with a free service to scan or display the vulnerabilities in container images.

The results of container image scans are stored in a GitLab Repository.

When the user asks to show the results from an image, the frontend consumes the GitLab API to retrieve the file with vulns from this image. In case this image is not scanned yet, the user has the option to schedule a scan using a google form.

When this form is filled, the data is sent to a Google Spreadsheet.

A GitHub Workflow runs every 5 minutes to check if there are new answers in this Spreadsheet. For each new image in the Spreadsheet, this workflow triggers another Workflow to scan the image and save the result in the GitLab Repository.

Why store in GitLab?

GitLab provides bigger limits.

Here's a summary of differences in offering on public cloud and free tier:

	Free users	Max repo size (GB)	Max file size (MB)	Max API calls per hour (per client)
GitHub	3	2	100	5000
BitBucket	5	1	Unlimited (up to repo size)	5000
GitLab	Unlimited	10	Unlimited (up to repo size)	36000

Google Drive

This choice was a "quick win". In a usual deployment, the backend could call an API passing secrets without the clients knowing the secrets.

But because I am using GitHub Pages I cannot use that(Well, I could do it in the javascript, but anyone using the Browser Inspect would see the secrets. So let's don't do it 😉)

This makes the Google Spreadsheet perform as a Queue.

Google Form:

Google Spreadsheet with answers:

GitHub Workflows

The Schedule Workflow runs at most every 5 minutes. This workflow executes the python script that checks if there are new rows in the Google Spreadsheet, and for each row is made an HTTP request to trigger the event repository_dispatch.

This makes the workflows perform as backend workers.

Schedule in workflow:

name: Schedule
on:
  schedule:
    - cron:  '*/5 * * * *'
...

Event repository_dispatch in WorkFlow:

name: Report
on: [repository_dispatch]
...

Screenshots:

Schedule History:

Schedule WorkFlow:

Scans History:

Report Workflow:

Scan Report stored in GitLab:

Source Code:

Do you want to know more about GitHub Actions?

Github Pages

The Frontend is running in GitHub Pages.

By default an application running in GH Pages is hosted as http://<github-user>.github.io/<repository>.

But GitHub allows you to customize the domain, because that it's possible to access Vilicus using https://vilicus.edersonbrilhante.com.br instead of http://edersonbrilhante.github.io/vilicus.

GitHub Workflow to build the application and deploy it in GH Pages

Building the source code:

- name: Build
  run: |
    cd website
    npm install
    npm run-script build
  env:
    REACT_APP_GA_CODE: ${{ secrets.REACT_APP_GA_CODE }}
    REACT_APP_FORM_SCAN: ${{ secrets.REACT_APP_FORM_SCAN }}

Deploying the build:

- name: Deploy
  uses: JamesIves/github-pages-deploy-action@releases/v3
  with:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    BRANCH: gh-pages
    FOLDER: website/build

Source Code:

Do you want to know more about GitHub Pages?

That’s it!

In case you have any questions, please leave a comment here or ping me on LinkedIn.

Fast startup application with database stored in container images

Ederson Brilhante — Thu, 01 Apr 2021 15:15:54 +0000

TL;DR; This article shows which strategy I implemented to allow an application to be ready to use in a few minutes rather than many hours.

In this article, I will talk about the strategy I used in the project Vilicus to have big databases synced in new setups. For those who don't know Vilicus yet, I recommend reading my article about it.

Why the application takes too much time to start?

At this moment the project Vilicus uses Anchore, Clair, and Trivy as vendors to run security scans in container images. Each vendor has its own programming language, database, internal dependencies and can use different vulnerabilities databases.

Vilicus itself starts in milliseconds, but to be ready to use it's necessary to wait for the vendors to sync the vulnerabilities database with the latest changes. But these syncs can consume a lot of time.

See for example Anchore, the one with more time-consuming to complete the sync:

There is no exact time frame for the initial sync to complete as it depends heavily on environmental factors, such as the host's memory/cpu allocation, disk space, and network bandwidth. Generally, the initial sync should complete within 8 hours but may take longer. Subsequent feed updates are much faster as only deltas are updated.
https://docs.anchore.com/current/docs/faq/

Clair takes more or less 20 minutes. And Trivy is ready in a few seconds.

If you run everything from scratch will take almost 1 day to sync all vulnerabilities databases, but after this major sync, the next syncs will be faster.

This will be a problem if you would like to run an ephemeral instance in your CI / CD, so waiting hours for the sync to be completed before you can run the first scan will be inviable. Thinking about how to fix this problem, I came with a solution: Save updated database snapshots in container images every day.

Now you must be thinking, this is not a good practice, and normally I would agree. But I believe there are exceptions in specific cases, such as fixing the problem is more important than conventions.

Saving the database in a container image

I'll show you in detail how I made Anchore work, but Clair and Trivy are not much different

Anchore

First I have a compacted dump SQL, with the database already synced with less last 6 months, stored in a container image: vilicus/anchoredb:dumpsql. So we don't need to wait many hours, we just update the delta.

I used this image as a base to create a local image(vilicus/anchoredb:files) with a script to restore the database when this image runs as a container.

Dockerfile content

FROM vilicus/anchoredb:dumpsql as dumpsql

FROM postgres:9.6.21-alpine
LABEL vilicus.app.version=9.6.21-alpine

COPY --chown=postgres:postgres --from=dumpsql /opt/vilicus/data/anchore_db.tar.gz /opt/vilicus/data/anchore_db.tar.gz
COPY deployments/dockerfiles/anchore/db/files/restore-dbs.sh /docker-entrypoint-initdb.d/01.restore-dbs.sh

Building the container image

docker build -f deployments/dockerfiles/anchore/db/files/Dockerfile -t vilicus/anchoredb:files .

The image vilicus/anchoredb:files is referenced in deployments/docker-compose.updater.yml

Here we start the anchore and the anchoredb.

docker-compose -f deployments/docker-compose.updater.yml up \
    --build -d --force \
    --remove-orphans \
    --renew-anon-volumes anchore

After that, we run this command to restore the database.

docker exec anchoredb sh -c 'docker-entrypoint.sh postgres' &

So we wait for the restore and the database we ready to be connected.

docker run --network container:anchore vilicus/vilicus:latest \
    sh -c "dockerize -wait http://anchore:8228/health -wait-retry-interval 10s -timeout 1000s echo done"

With the Anchore Engine and the DB ready, we start the sync.

docker exec anchore sh -c 'anchore-cli system wait'

When the sync finishes we stop anchore and we kill gracefully the Postgres PID in anchoredb.

docker stop anchore
docker exec -u postgres anchoredb sh -c 'pg_ctl stop -m smart'

We commit the container, with the changes made by the sync, into a new container image vilicus/anchoredb:local-update

CID=$(docker inspect --format="{{.Id}}" anchoredb)
docker commit $CID vilicus/anchoredb:local-update

So we finally build the container image that goes to docker hub, by copying the Postgres data from the image vilicus/anchoredb:local-update

Dockerfile content

FROM as db
FROM postgres:9.6.21-alpine
COPY --chown=postgres:postgres --from=db /data/ /data

Building the container image

docker build -f deployments/dockerfiles/anchore/db/Dockerfile -t vilicus/anchoredb:latest .

Check the complete script here

Clair and Trivy

For Clair check here.

For Trivy check here.

Updating the images every day

To have the databases with the latest changes, I have a GitHub workflow that runs a job everyday building the images and pushing them to the Docker Hub.

Check the workflow

Complete workflow

That's it!

In case you have any questions, please leave a comment here or ping me on 🔗 LinkedIn.

GitLab Runners as a Service with Github Action

Ederson Brilhante — Thu, 01 Apr 2021 14:52:43 +0000

TL;DR; This article will show how to implement the action "Gitlab Runner Service Action" in a "GitHub Workflow" that is triggered by a "GitLab-CI job", and this way having temporary GitLab Runners hosted by GitHub.

For more info about GitHub workflow, check the official documentation

For more info about GitLab-CI, check the official documentation

Steps

Step 1

Create a new GitHub repository with the following GitHub Workflow. File location: .github/workflows/gitlab-runner.yaml

name: Gitlab Runner Service
on: [repository_dispatch]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Maximize Build Space
        uses: easimon/maximize-build-space@master
        with:
          root-reserve-mb: 512
          swap-size-mb: 1024
          remove-dotnet: 'true'
          remove-android: 'true'
          remove-haskell: 'true'

      - name: Gitlab Runner
        uses: edersonbrilhante/gitlab-runner-action@main
        with:
          registration-token: "${{ github.event.client_payload.registration_token }}"
          docker-image: "docker:19.03.12"
          name: ${{ github.run_id }}
          tag-list: "crosscicd"

What does this workflow do?

This workflow will run just when the event repository_dispatch is triggered. The first step will be to increase the free space removing useless packages for our GitLab runner. And the second step will run the action that registers a new GitLab Runner with a tag crosscicd, so start it and unregister it after a GitLab-CI job is completed with success or failure.

Step 2

Create a new GitLab repository with the following GitLab-CI config. File location: .gitlab-ci.yml

start-crosscicd:
  image: alpine
  before_script:
    - apk add --update curl && rm -rf /var/cache/apk/*
  script: |
    curl -H "Authorization: token ${GITHUB_TOKEN}" \
    -H 'Accept: application/vnd.github.everest-preview+json' \
    "https://api.github.com/repos/${GITHUB_REPO}/dispatches" \
    -d '{"event_type": "gitlab_trigger_'${CI_PIPELINE_ID}'", "client_payload": {"registration_token": "'${GITLAB_REGISTRATION_TOKEN}'"}}'

github:
  image: docker:latest
  services:
    - name: docker:dind
      alias: thedockerhost
  variables:
    DOCKER_HOST: tcp://thedockerhost:2375/
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  script:
    - df -h
    - docker run --privileged ubuntu df -h
  tags:
    - crosscicd

What does this gitlab-ci?

The job start-crosscicd will trigger the GitHub workflow, creating the GitLab runner with the tag crosscicd. And the job GitHub will wait for a runner with a tag crosscicd.

Step 3

Set the EnvVars in the new GitLab Repo

    GITHUB_REPO:<username>/<github-repo>
    GITHUB_TOKEN:<GitHub Access Token>
    GITLAB_REGISTRATION_TOKEN:<GitLab Registration Token>

How to create a new GitHub Access Token:

Go to https://github.com/settings/tokens/new
Mark the item workflow and click in generate a token

How to get Registration Token:

Go to https://gitlab.com/<username>/<repo>/-/settings/ci_cd and click and expand Runners
Copy the Registration Token

Where to store the EnvVars?

Go to https://gitlab.com/<username>/<repo>/-/settings/ci_cd and click and expand Variables
Click in Add Variable and save it for each EnvVar

Step 4

Now your pipeline is ready to run the GitLab Runner in GitHub trigger by Gitlab-CI Job :)

Example

Video Demo

Screenshots

Job start-crosscicd trigger Github Workflow

Workflow triggered by Gitlab-CI job

There is 17GB free by default

After Maximize we have 54GB free to use

Code

That’s it!

In case you have any questions, please leave a comment here or ping me on 🔗 LinkedIn.

Vilicus — An overseer for security scanning of container images

Ederson Brilhante — Wed, 31 Mar 2021 20:19:18 +0000

Vilicus is an open-source tool that orchestrates security scans of container images(Docker/OCI) and centralizes all results into a database for further analysis and metrics.

Why scan for vulnerabilities?

A recent analysis of around 4 million Docker Hub images by cyber security firm Prevasio found that 51% of the images had exploitable vulnerabilities. A large number of these were cryptocurrency miners, both open and hidden, and 6,432 of the images had malware.
https://www.infoq.com/news/2020/12/dockerhub-image-vulnerabilities/

Image from https://prevasio.com/static/web/viewer.html?file=/static/Red_Kangaroo.pdf

Docker image security scanning is a process for finding security vulnerabilities within your Docker image files.
Typically, image scanning works by parsing through the packages or other dependencies that are defined in a container image file, then checking to see whether there are any known vulnerabilities in those packages or dependencies.
https://resources.whitesourcesoftware.com/blog-whitesource/docker-image-security-scanning

How does it work?

There are many tools to scan container images for vulnerabilities such as Anchore, Clair, and Trivy. But sometimes the results from the same image can be different. And this project comes to help the developers to improve the quality of their container images by finding vulnerabilities and thus addressing them with agnostic sight from vendors.

Some articles comparing the scanning tools:

Architecture

Cached Database

Vilicus updates daily the vendor databases with the latest changes in the vulns DBs.

Using a strategy to storage the database data in layers of docker images, the whole platform is ready to use in minutes instead of hours. Starting the sync feed with vulns from scratch can take at least 6 hours.

Check the strategy used in Anchore, Clair and Trivy

Local Registry

Vilicus provides a local registry, so you can build a local image and scanning it without pushing it to a remote repository.

docker build -t localhost:5000/local-image:my-tag .

curl -o docker-compose.yml https://raw.githubusercontent.com/edersonbrilhante/vilicus/main/deployments/docker-compose.yml

docker-compose up -d

IMAGE=localregistry.vilicus.svc:5000/local-image:my-tag

docker run -v ${PWD}/artifacts:/artifacts \
  --network container:vilicus \
  vilicus/vilicus:latest \
  sh -c "dockerize -wait http://vilicus:8080/healthz -wait-retry-interval 60s -timeout 2000s vilicus-client -p /opt/vilicus/configs/conf.yaml -i ${IMAGE}  -t /opt/vilicus/contrib/sarif.tpl -o /artifacts/results.sarif"

GitHub Action

GitHub Actions makes it easy to automate all your software workflows, now with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.
https://github.com/features/actions

Vilicus provides a GitHub action to help you scanning container images in your CI/CD.

Container scanning

A scan can be done using a remote image and a local image. Using a remote repository such as docker.io the image will be docker.io/your-organization/image:tag:

  - name: Scan image
    uses: edersonbrilhante/vilicus-github-action@main
    with:
      image: "docker.io/myorganization/myimage:tag"

And to use a local image its need to tag as localhost:5000/image:tag:

  - name: Scan image
    uses: edersonbrilhante/vilicus-github-action@main
    with:
      image: "localhost:5000/myimage:tag"

Full example

Complete example with steps for cleaning space, building local image, Vilicus scanning, and uploading results to GitHub Security

name: Container Image CI
on: [push]
jobs:
  build
    runs-on: ubuntu-latest
    steps:
      - name: Maximize Build Space
        uses: easimon/maximize-build-space@master
        with:
          root-reserve-mb: 512
          swap-size-mb: 1024
          remove-dotnet: 'true'
          remove-android: 'true'
          remove-haskell: 'true'
      - name: Checkout branch
        uses: actions/checkout@v2
      - name: Build the Container image
        run: docker build -t localhost:5000/local-image:${GITHUB_SHA} .
      - name: Vilicus Scan
        uses: edersonbrilhante/vilicus-github-action@main
        with:
          image: localhost:5000/local-image:${{ github.sha }}
      - name: Upload results to github security
        uses: github/codeql-action/upload-sarif@v1
        with:
          sarif_file: artifacts/results.sarif

Results in GitHub Security

Check an example using Vilicus GitHub Action

Pipeline example

List with all vulns found

Vuln details

Source Code

VIlicus GitHub Action

Vilicus

That’s it!

In case you have any questions, please leave a comment here or ping me on 🔗 LinkedIn.