DEV Community: Daniel Quackenbush

AzureAD 3.7: Dynamic credentials with Flexible Federated Identity

Daniel Quackenbush — Wed, 19 Nov 2025 19:25:27 +0000

HashiCorp’s AzureAD provider introduces a new resource in the 3.7 release: azuread_application_flexible_federated_identity_credential. This unlocks dynamic credentials using flexible claim matching, so you can scope access to specific runs, workspaces, or phases for terraform cloud.

What’s new

Flexible claim matching: Match on claims like sub using expressions (e.g., by organization, project, workspace, and run phase).
Wildcard TFC integration: Ability to wildcard within the sub.

Quick walkthrough (HCP Terraform)

The snippet below creates:

An Azure AD application and service principal
A subscription‑scoped role assignment
A TFC workspace to consume the credentials

Setup Azure Application

terraform {
  required_providers {
    azuread = {
      source  = "hashicorp/azuread"
      version = ">= 3.7.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.0.0"
    }
    tfe = {
      source  = "hashicorp/tfe"
      version = ">= 0.53.0"
    }
  }
}

provider "azurerm" {
  features {}
}

data "azuread_client_config" "current" {}
data "azurerm_subscription" "current" {}

resource "azuread_application" "tfc_application" {
  display_name = "tfc-application"
  owners       = [data.azuread_client_config.current.object_id]
}

resource "azuread_service_principal" "tfc_service_principal" {
  client_id = azuread_application.tfc_application.client_id
}

resource "azurerm_role_assignment" "tfc_role_assignment" {
  scope                = data.azurerm_subscription.current.id
  principal_id         = azuread_service_principal.tfc_service_principal.object_id
  role_definition_name = "Contributor"
}

variable "tfe_organization" {
  type = string
}

data "tfe_project" "default" {
  name         = "Default Project"
  organization = var.tfe_organization
}

resource "tfe_workspace" "team_1" {
  name         = "workspace-team-1"
  project_id   = data.tfe_project.default.id
  organization = var.tfe_organization
}

Enable variables in your TFC workspace

Set the required environment variables so that runs authenticate via Workload Identity:

resource "tfe_variable" "enable_azure_provider_auth" {
  workspace_id = tfe_workspace.team_1.id

  key         = "TFC_AZURE_PROVIDER_AUTH"
  value       = "true"
  category    = "env"
  description = "Enable the Workload Identity integration for Azure."
}

resource "tfe_variable" "tfc_azure_client_id" {
  workspace_id = tfe_workspace.team_1.id

  key         = "TFC_AZURE_RUN_CLIENT_ID"
  value       = azuread_application.tfc_application.id
  category    = "env"
  description = "The Azure Client ID runs will use to authenticate."
}

# Optional: override the default audience used in tokens
resource "tfe_variable" "tfc_azure_audience" {
  workspace_id = tfe_workspace.team_1.id

  key         = "TFC_AZURE_WORKLOAD_IDENTITY_AUDIENCE"
  value       = "api://AzureADTokenExchange"
  category    = "env"
  description = "The value to use as the audience claim in run identity tokens."
}

resource "tfe_variable" "arm_subscription_id" {
  workspace_id    = tfe_workspace.team_1.id

  key             = "ARM_SUBSCRIPTION_ID"
  value           = data.azurerm_subscription.current.subscription_id
  category        = "env"
}

resource "tfe_variable" "arm_tenant_id" {
  workspace_id = tfe_workspace.team_1.id

  key             = "ARM_TENANT_ID"
  value           = data.azurerm_subscription.current.tenant_id
  category        = "env"
}

Configure single flexible credential (wildcard)

Historically, Terraform Cloud runs often required separate federated identity credentials per run_phase (for example, one for plan and another for apply). With flexible federated identity credentials, you can reduce this management overhead by using the restricted expression language to wildcard the sub claim and authorize multiple phases under a single credential using the matches operator. See Microsoft’s guidance for supported issuers and operators. Microsoft Entra flexible federated identity credentials (preview)

Before: per‑phase credentials

resource "azuread_application_federated_identity_credential" "tfc_federated_credential_plan" {
  application_id = azuread_application.tfc_application.id
  display_name   = "tfc-federated-credential-plan"
  audiences      = ["api://AzureADTokenExchange"]
  issuer         = "https://app.terraform.io"
  subject        = "organization:${var.tfe_organization}:project:${data.tfe_project.default.name}:workspace:${tfe_workspace.team_1.name}:run_phase:plan"
}

resource "azuread_application_federated_identity_credential" "tfc_federated_credential_apply" {
  application_id = azuread_application.tfc_application.id
  display_name   = "tfc-federated-credential-apply"
  audiences      = ["api://AzureADTokenExchange"]
  issuer         = "https://app.terraform.io"
  subject        = "organization:${var.tfe_organization}:project:${data.tfe_project.default.name}:workspace:${tfe_workspace.team_1.name}:run_phase:apply"
}

After: Single Wildcarded Credential

resource "azuread_application_flexible_federated_identity_credential" "plan_and_apply" {
  application_id             = azuread_application.tfc_application.id
  display_name               = "tfc-federated-flexible-credential"
  audience                   = "api://AzureADTokenExchange"
  issuer                     = "https://app.terraform.io"
  claims_matching_expression = "claims['sub'] matches 'organization:${var.tfe_organization}:project:${data.tfe_project.default.name}:workspace:${tfe_workspace.team_1.name}:run_phase:*'"
}

Deploying an RKE2 Cluster with CAPI, Rancher Turtles, and Proxmox

Daniel Quackenbush — Mon, 19 May 2025 13:39:20 +0000

If you’ve recently set up Rancher and want to deploy your first Kubernetes cluster on Proxmox, this guide will walk you through the process. By combining Cluster API (CAPI), Rancher Turtles, and Proxmox, you can build and manage clusters directly from Rancher. CAPI enables declarative management of Kubernetes clusters, while Rancher Turtles integrates Rancher’s powerful management features with Proxmox as your infrastructure provider.

Understanding the Components

Cluster API (CAPI): A Kubernetes sub-project that provides declarative APIs for cluster creation, configuration, and management.
Rancher Turtles: An extension to CAPI that enhances its functionality, particularly for managing RKE2 clusters.
Proxmox: An open-source virtualization platform that you will use as your infrastructure provider.
RKE2: A Kubernetes distribution focused on security and compliance.

Prerequisites

Before you begin, ensure you have the following:

A Proxmox VE template: An image to clone for your cluster nodes.
A Rancher instance: Rancher should be installed and accessible.

Auto-Importing Your Cluster into Rancher Manager

To simplify cluster management, configure Rancher Manager to automatically import your new CAPI cluster. You can:

Import all CAPI clusters in a namespace:

Label the namespace containing your clusters:

kubectl label namespace default cluster-api.cattle.io/rancher-auto-import=true

Import a specific cluster
Label an individual cluster:

kubectl label cluster.cluster.x-k8s.io my-cluster cluster-api.cattle.io/rancher-auto-import=true

By applying one of these labels, your newly created CAPI cluster will be automatically imported into Rancher Manager, allowing you to manage it alongside your other Kubernetes clusters.

Installation

Setting Up Proxmox User and API Token

Use a dedicated Proxmox VE user with an API token. You can create this through the UI or by running the following commands on the Proxmox VE node:

pveum user add capmox@pve
pveum aclmod / -user capmox@pve -role PVEAdmin
pveum user token add capmox@pve capi -privsep 0

These commands create a user named capmox, assign the PVEAdmin role, and create an API token for this user.

Install Rancher Turtles

helm repo add rancher-turtles https://rancher-sandbox.github.io/rancher-turtles/
helm repo update helm install rancher-turtles rancher-turtles/rancher-turtles -n cattle-system --create-namespace

Install IPAM Components

Deploy the IP Address Management (IPAM) components. Cluster API IP Address Management (IPAM) provider allows clusters to manage pools of IP addresses using Kubernetes-native resources:

kubectl apply -f https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster/releases/download/v1.0.0/ipam-components.yaml

Deploy Proxmox Infrastructure Components

Install the Proxmox-specific infrastructure components. The components contain

Custom Resource Definitions (CRDs): New Kubernetes resource types for Proxmox-specific infrastructure (like ProxmoxCluster, ProxmoxMachine, etc.).
Controller Deployment: A controller (operator) that watches for Cluster API resources and provisions/manages Proxmox VMs accordingly.
RBAC Roles and Bindings Permissions so the controller can manage resources securely.

Download the Proxmox infrastructure components YAML and update the following values before applying:

curl -LO https://github.com/ionos-cloud/cluster-api-provider-proxmox/releases/latest/download/infrastructure-components.yaml

Edit the file to set your Proxmox credentials:

capmox-manager-credentials:   
   secret: ${PROXMOX_SECRET=""}  
   token: ${PROXMOX_TOKEN=""}  
   url: ${PROXMOX_URL=""}

Then apply the YAML:

kubectl apply -f infrastructure-components.yaml

Cluster Configuration

Now, let's look at the key parts of the cluster configuration:

1. Define the Cluster

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: my-cluster
  namespace: default
spec:
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: ProxmoxCluster
    name: my-cluster
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: RKE2ControlPlane
    name: my-cluster-control-plane
  clusterNetwork:
    pods:
      cidrBlocks: [192.168.0.0/22]

2. Configure the Proxmox Cluster

Modify the allowed Proxmox nodes, control plane endpoint, and IP configuration to match your Proxmox environment and network setup. The controlPlaneEndpoint should reference a loadbalancer that sends traffic to the control-plane nodes. If you don’t have a load balancer, consider deploying kube-vip in your RKE2ControlPlane manifests.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxCluster
metadata:
  name: my-cluster
  namespace: default
spec:
  allowedNodes:
    - proxmox1
    - proxmox2
    - proxmox3
  controlPlaneEndpoint:
    host: 192.168.12.219
    port: 6443
  dnsServers:
      - 192.168.12.22
      - 1.1.1.1
  ipv4Config:
    addresses:
      - 192.168.12.220-192.168.12.222
    prefix: 22
    gateway: 192.168.12.1

3. Define the Machine Template

This template defines how new machines should be created in Proxmox. Adjust the source node, template ID, storage, resource allocations, and network settings based on their available resources and requirements.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: ProxmoxMachineTemplate
metadata:
  name: my-cluster-control
  namespace: default
spec:
  template:
    spec:
      sourceNode: "proxmox1"
      templateID: 100
      full: true
      storage: ceph
      numCores: 2
      memoryMiB: 8192
      disks:
        bootVolume:
          sizeGb: 100
          disk: scsi0
      network:
        default:
          vlan: 200
          bridge: vmbr0
          model: virtio

4. Set Up the RKE2 Control Plane

Update the RKE2 version, adjust the number of replicas for high availability, and modify or add any custom configurations as needed. The manifest also includes a template for kube-vip; if you plan to deploy kube-vip, update the content section of /var/lib/rancher/rke2/server/manifests/kubevip.yaml accordingly.

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: RKE2ControlPlane
metadata:
  name: my-cluster-control-plane
  namespace: default
spec:
  version: v1.31.4+rke2r1
  replicas: 1
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: ProxmoxMachineTemplate
    name:  my-cluster-control
    namespace: default
  rolloutStrategy:
    type: RollingUpdate
  files:
    - path: "/var/lib/rancher/rke2/server/manifests/coredns-config.yaml"
      owner: "root:root"
      permissions: "0640"
      content: |
        apiVersion: helm.cattle.io/v1
        kind: HelmChartConfig
        metadata:
          name: rke2-coredns
          namespace: kube-system
        spec:
          valuesContent: |-
            tolerations:
            - key: "node.cloudprovider.kubernetes.io/uninitialized"
              value: "true"
              effect: "NoSchedule"
    - path: "/var/lib/rancher/rke2/server/manifests/kubevip.yaml"
      owner: "root:root"
      permissions: "0640"
      content: |
        # KubeVIP configuration...
  preRKE2Commands:
    - apt-get update && apt-get install -y qemu-guest-agent
    - systemctl enable qemu-guest-agent --now
  serverConfig:
    cni: cilium
  agentConfig:
    kubelet:
      extraArgs:
        - --cloud-provider=external
        - --provider-id=proxmox://{{ ds.meta_data.instance_id }}

Deploying Your Cluster

To deploy your cluster, save the entire YAML configuration to a file (e.g., my-cluster.yaml) and apply it using kubectl. This will initiate the cluster creation process. CAPI and Rancher Turtles will work together to provision the necessary resources in Proxmox and set up your RKE2 cluster.

kubectl apply -f my-cluster.yaml

Troubleshooting

If you encounter issues during deployment or management, check the logs of these system pods:

rancher-turtles-system: Manages Rancher Turtles integration.
rke2-bootstrap-system: Handles bootstrapping for RKE2 nodes.
rke2-control-plane-system: Manages RKE2 control plane operations.
capi-system: Runs core Cluster API controllers.
capmox-system: Contains Proxmox-specific CAPI provider controllers.
capi-ipam-in-cluster-system: Manages in-cluster IP address allocation.

Terraform Ephemeral Resources

Daniel Quackenbush — Wed, 27 Nov 2024 15:57:58 +0000

Storing sensitive data like database passwords and API keys in Terraform state files poses significant security risks. Terraform 1.10 introduced the ephemeral resource to address this issue, allowing dynamic retrieval of sensitive values during runtime without persisting them in the state file.

Resources

Ephemeral resources are defined using a new block type in HCL:

ephemeral "<resource type>" "<resource name>" {
  # attributes and nested blocks
}

These resources are referenced similarly to data sources but are not stored in the state file. Attributes and nested blocks will vary per provider implementation; however, they are referenced similarly to data sources.

Example: Using AWS Secrets Manager with Postgres

In this example, the database credentials are retrieved from AWS Secrets Manager and used to configure the PostgreSQL provider without being stored in the state file.

data "aws_secretsmanager_secret" "example" {
    name = "myprodsecret"
}

ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = data.aws_secretsmanager_secret.example.arn
}

locals {
  credentials = jsondecode(ephemeral.aws_secretsmanager_secret_version.db.secret_string)
}

provider "postgresql" {
  host     = "1.2.3.4"
  port     = "5432"
  username = local.credentials["username"]
  password = local.credentials["password"]
}

data "postgresql_tables" "my_tables" {
  database = "postgres"
}

output "tables" {
  value = data.postgresql_tables.my_tables.tables.*.object_name
}

If we examine the items in the state, there is no reference to aws_secretsmanager_secret_version.

# terraform state list
data.aws_secretsmanager_secret.example
data.postgresql_tables.my_tables

Supported Providers

At the time of writing, the current supported ephemeral blocks are:

kubernetes_certificate_signing_request
kubernetes_token_request
aws_secretsmanager_secret_version
aws_lambda_invocation
aws_kms_secrets
azurerm_key_vault_secret
azurerm_key_vault_certificate
google_service_account_access_token
google_service_account_id_token
google_service_account_jwt
google_service_account_key

Variables and Outputs

Variables can be declared as ephemeral with ephemeral = true.

variable "api_key" {
  type        = string
  ephemeral   = true
  description = "an api key not stored in state"
}

Outputs in non-root modules can also be declared as ephemeral:

Example: Using Ephemeral Outputs in Modules

output "database_secret" {
  ephemeral = true
  value     = aws_secretsmanager_secret_version.db.secret_string
}

module "database" {
  source = "./database"
}

locals {
  credentials = jsondecode(module.database.database_secret)
}

provider "postgresql" {
  host     = data.aws_db_instance.example.address
  port     = data.aws_db_instance.example.port
  username = local.credentials["username"]
  password = local.credentials["password"]
}

Looking at the state again, there is still no reference to aws_secretsmanager_secret_version:

# terraform state list
data.postgresql_tables.my_tables
module.database.data.aws_secretsmanager_secret.example

Benefits of Ephemeral Resources

Enhanced Security: Sensitive data is not stored in state files, reducing the risk of unauthorized access.
Dynamic Retrieval: Values are fetched at runtime, ensuring up-to-date information.
Compliance: Helps meet regulatory requirements by keeping secrets out of long-term storage.
Simplified Secret Management: Integrates with secret management solutions like AWS Secrets Manager.

Terraform's ephemeral feature significantly improves security posture by keeping sensitive data out of state files. By leveraging this feature, teams can maintain the benefits of infrastructure as code while adhering to security best practices. For more detailed information, refer to the Hashicorp documentation on ephemeral resources.

Developing a Nomad Autoscaler for Harvester

Daniel Quackenbush — Tue, 28 May 2024 12:38:35 +0000

Nomad orchestrates application deployment and management. As applications grow in size, managing resource consumption becomes crucial. The Nomad Autoscaler is a pluggable service that makes workload scaling more accessible, empowering users to create logic for scaling their infrastructure.

Developing a custom plugin is especially beneficial when catering to cloud environments or hypervisors that aren't supported by the HashiCorp community. This blog will guide you through creating a Nomad Autoscaler plugin, through the use of the exposed methods: SetConfig, Scale, and Status.

Defining the Plugin Struct

For our Nomad Autoscaler plugin, we'll define a struct to hold configuration and other state information for scaling on Harvester. The Plugin struct implements the sdk.Target interface to work as a nomad autoscaling plugin. The Plugin struct should contain all the state needed to actually implement autoscaling, such as configuration. loggers, and api clients.

package main

import (
    "context"
    "fmt"
    "time"
    "github.com/hashicorp/go-hclog"
    "github.com/drewmullen/harvester-go-sdk"
    "github.com/hashicorp/nomad/api"
)

type HarvesterPlugin struct {
    config          map[string]string
    logger          hclog.Logger
    HarvesterClient *harvester.APIClient
    NomadClient     *api.Client
    // Additional Config
}

func NewPlugin(log hclog.Logger) *HarvesterPlugin {
    return &HarvesterPlugin{
        logger: log,
    }
}

Configuring the Plugin

The target plugin contains two different config parameters:

target: what to scale
policy: when to scale

Target Configuration

The target configuration contains the plugin-specific configuration, containing authentication credentials, global settings, etc.

target {
    driver = "harvester"
    config = {
       harvester_url = "https://harvester.example.com"
       auth_token    = "eyabc123"
    }
}

Once instantiated, the Nomad autoscaler service will pass the target configuration options to the plugin's SetConfig method, which can then be used to set up the plugin fields. The configuration will also contain options as defined in the General Options documentation.

A sample setup might look something like this:

func (hp *HarvesterPlugin) SetConfig(config map[string]string) error {
    token := getEnvOrConfig("HARVESTER_TOKEN", config, configKeyAuthToken) // A function that returns in order of priority environment var, config value.
    url := getEnvOrConfig("HARVESTER_URL", config, configKeyHarvesterURL) // configKeyHarvesterURL is a const defined elsewhere
    hp.HarvesterClient = harvester.NewAPIClient(&harvester.Configuration{
        DefaultHeader: map[string]string{"Authorization": "Bearer " + token},
        UserAgent: "nomad-autoscaler",
        Debug:     false,
        Servers: harvester.ServerConfigurations{
           {URL: url, Description: "Harvester API Server"},
        },
    })

    apiConfig := &api.Config{
        Address:    config["nomad_address"],
        Region:     config["nomad_region"],
        Namespace:  config["nomad_namespace"],
    }
    if token, ok := config["nomad_token"]; ok {
        apiConfig.Headers = map[string][]string{"X-Nomad-Token": []string{token}}
    }
    nomadClient, err := api.NewClient(apiConfig)
    if err != nil {
        return fmt.Errorf("failed to create Nomad client: %v", err)
    }
    hp.NomadClient = nomadClient
    // Any other additional Config
    return nil
}

Scaling

Cluster Operators author scaling policing when interacting with the Autoscaler. The config provided is then pass as a parameter to the Scale method to dynamically allocate the necessary resources.

scaling "cluster_policy" {
  enabled = true
  min     = 1 # min number of VMs to scale
  max     = 2 # max number of VMs to scale

  policy {
    ....
    target "aws-asg" {
      dry-run            = "false"
      node_class         = "linux"
      node_group_name    = "nomad"
      namespace          = "default"
      cpu_request        = "2"
      memory_request     = "4Gi"
      ...
    }
  }
}

With the configuration defined, Nomad passes the scaling config to the plugin's Scale method. Your hypervisor will determine the actual implementations of how to calculate the active nodes for scale operations.

func (hp *HarvesterPlugin) Scale(action sdk.ScalingAction, config map[string]string) error {
    // config parsing removed for simplicity
    ctx, cancel := context.WithTimeout(context.Background(), hp.scaleTimeout)
    defer cancel()

    total, _, remoteIDs, err := hp.countReady(ctx, nodeGroup, namespace)
    if err != nil {
        return fmt.Errorf("failed to count servers in harvester: %v", err)
    }

    diff, direction := hp.calculateDirection(total, action.Count)
    switch direction {
    // SCALE_IN is an enum utilized as output of the calculateDirection function
    case SCALE_IN:
        if err := hp.scaleIn(ctx, diff, remoteIDs, config); err != nil {
            return fmt.Errorf("failed to perform in: %v", err)
        }
    // SCALE_OUT is an enum utilized as output of the calculateDirection function
    case SCALE_OUT:
        if err := hp.scaleOut(ctx, diff, config); err != nil {
            return fmt.Errorf("failed to perform out: %v", err)
        }
    default:
        hp.logger.Debug("scaling not required", "node group", nodeGroup, "current_count", total, "strategy_count", action.Count)
        return nil
    }
    return err
}

Draining Nodes

During the scaleIn method, HashiCorp recommends that you first drain the node. Draining and purging nodes is critical to scaling down operations, providing reliability to allow for applications to gracefully shutdown. After some time, with the node offline, Nomad's garbage collector will then remove the node from the cluster.

func (hp *HarvesterPlugin) drainNode(ctx context.Context, nodeID string, timeout time.Duration) error {
    _, err := hp.NomadClient.Nodes().UpdateDrainOpts(
        nodeID,
        &api.DrainOptions{
            DrainSpec: &api.DrainSpec{
                Deadline:         timeout,
                IgnoreSystemJobs: true,
            },
            MarkEligible: false,
        },
        nil,
    )
    if err != nil {
        hp.logger.Warn(fmt.Sprintf("Failed to drain %v. Will continue to deleting: %v", nodeID, err))
    } else {
        drainCtx, cancel := context.WithTimeout(ctx, timeout)
        defer cancel()
        err := hp.waitForDrained(drainCtx, nodeID)
        if err != nil {
            hp.logger.Warn(fmt.Sprintf("Failed to drain %v: %v", nodeID, err))
        }
    }
}

Status

The Status function reports the current status of your plugin, which helps debug and monitor purposes. The plugin determines the current running count using information returned by the plugin's Status method. The method returns an sdk.TargetStatus to determine if the next Scale function can be performed, as well as the current running count to determine the next strategy calculation.

func (hp *HarvesterPlugin) Status(config map[string]string) (*sdk.TargetStatus, error) {
    total, active, _, err := hp.countReady(context.Background(), nodeGroup, namespace)
    if err != nil {
        return nil, fmt.Errorf("failed to count Harvester servers: %v", err)
    }
    return &sdk.TargetStatus{
        Ready: active == total,
        Count: total,
        Meta:  make(map[string]string),
    }, nil
}

Conclusion

Developing a Nomad Autoscaler plugin involves implementing key functions like SetConfig, Scale, and Status. Configuring the plugin requires defining target and policy blocks, which dictate what to scale and under what conditions. The proper handling of draining allows for control over the scaling, maintaining the reliability of your application.

Writing an autoscaler plugin lets you to tailor your hypervisor to the needs of your Nomad-managed infrastructure. Finally, here's a demo of the autoscaler in action.

For more details and examples, check out the Nomad Autoscaler Plugin authoring guide, and the Nomad Autoscaling tools documentation.

Special thanks to Steve Kalt for helping review this post.

Multi-Cluster Prometheus: Scaling Metrics Across Kubernetes Clusters

Daniel Quackenbush — Fri, 26 Jan 2024 20:11:03 +0000

Building upon Bartłomiej Płotka's insightful blog on Prometheus and its passthrough agent mode, this post dives into implementing multi-cluster Prometheus support. Notably, the official inclusion of support in the widely-used kube-prometheus-stack came with the release in July 2023, making it easier to extend Prometheus monitoring across clusters.

Helm Configuration: Connecting Global and Edge Clusters

To deploy a Prometheus agent in your Kubernetes cluster, use the kube-prometheus-stack chart with Helm installed and configured on your cluster.

Global Cluster Configuration

Update and apply global cluster values to enable remote metrics reception:

prometheus:
  prometheusSpec:
    enableRemoteWriteReceiver: true
    enableFeatures:
    - remote-write-receiver

Edge Cluster(s) Configuration

Update and apply edge cluster values to enable remote metric writing:

Specify the global cluster's ${hostname} remote write endpoint.
Include a scrape configuration to identify pods annotated with prometheus.io/scrape.
Assign a unique ${name} to distinguish metric results.

prometheus:
  agentMode: true
  prometheusSpec:
    remoteWrite:
      - name: ${name}
        url: https://${hostname}/api/v1/write
     # Add additional remote write configurations if needed
    additionalScrapeConfigs:
      - job_name: 'kubernetes'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - target_label: cluster
            replacement: ${name}

These resulting metrics are illustrated within the global Prometheus cluster, featuring an Apache web service managing ingress traffic in a cluster labeled as "edge."

Troubleshooting

Prometheus exposes internal metrics for effective troubleshooting. For in-depth troubleshooting steps, refer to the detailed Grafana Labs blog post.

Key metrics to monitor:

Queue Tracking: prometheus_remote_storage_highest_timestamp_in_seconds and prometheus_remote_storage_queue_highest_sent_timestamp_seconds will track the backlog/queue.
Shard Considerations: prometheus_remote_storage_shards_desired should be less than prometheus_remote_storage_shards_max. If it is greater, you may want to consider updating max_shards - see Prometheus Parameter Tuning.

Creating Function as a Service in Kubernetes with Argo Events

Daniel Quackenbush — Wed, 10 Jan 2024 22:31:48 +0000

Containers have changed service creation in software development, and with the advent of AWS Lambda, Function as a Service (FaaS) emerged, further reshaping the approach to service execution. However, within Kubernetes, the seamless integration of single-event processing posed a challenge for teams striving to harness the advantages without the burden of constructing complete services.

Argo Workflows + Argo Events is one solution in the domain that operates natively in Kubernetes, providing a robust orchestration framework. One of its key features, Sensors, acts as the crucial link between external systems and Kubernetes environments. Leveraging these sensors to subscribe to an SQS queue facilitates a seamless connection between public cloud infrastructure and Kubernetes. This integration empowers users to establish effortless data and action exchanges, enhancing system adaptability and scalability.

Exploring an SQS Workflow

One way to test out Argo Events + Workflows in action is by exploring how SQS Messages can be processed. Leveraging sensors to subscribe to an AWS Simple Queue Service (SQS) queue establishes a seamless connection between public cloud infrastructure and Kubernetes. Messages placed on a queue will multiplex, processing that data across functions to transform the data and then insert a record into my datastore. Since the queue volume is variable, having a functional process allows for both horizontal scale and for the application to be zero.

Argo Components

Argo Events + Argo Workflows have a few CRDs to create the message processing to the pod. The relationship between the two is Argo Events will provision and listen for events from the given source and then use Argo Workflows to execute. Where Argo Events' power comes from is that it can also invoke other triggers, including custom, but this post won't go into those specifics.

EventBus

The event bus is the transport layer from the Event to the Sensor.

apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
  namespace: argocd
spec:
  jetstream:
    version: latest

EventSource

Within Kubernetes, configuring a Service Account with IRSA allows us to subscribe to the specific queue. Within AWS, a queue-subscriber role has permission to review messages on the queue.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-queue
  namespace: argocd
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::0123456789:role/queue-subscriber

The event source is defined with the name, as well as the associated service account. In this example, an event source is being provisioned to listen for all messages on sqs. When the queue listener pod spins up, it will use the assumed role to listen for messages.

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: aws-sqs
  namespace: argocd
spec:
  template:
    serviceAccountName: test-queue
  sqs:
    test_queue:
      region: "us-east-1"
      queue: "test-queue"
      waitTimeSeconds: 20

Service Account/RBAC

The Sensor Workflow will require an operational role to create workflows. To provision, you can apply the example sensor yaml

Sensor

A sensor connects the EventSource through the EventBus to ultimately the destination of choice. In the case of Argo Workflows, the triggers definition will define a container for which to invoke.
The aws-sqs sensor resource contains multiple parts. The first is the dependency section, which we will reference later on.

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: aws-sqs
  namespace: argocd
spec:
  template:
    serviceAccountName: operate-workflow-sa
  dependencies:
    - name: test-dep
      eventSourceName: aws-sqs
      eventName: test_queue

The structure of an SQS message is essential because we may want to pass a few fields to our workflow later on.

{
    "context": {
        "type": "type_of_event_source",
        "specversion": "cloud_events_version",
        "source": "name_of_the_event_source",
        "id": "unique_event_id",
        "time": "event_time",
        "datacontenttype": "type_of_data",
        "subject": "name_of_the_configuration_within_event_source"
    },
    "data": {
        "messageId": "message id",
        "messageAttributes": "message attributes", 
        "body": "Body is the message data",
    }
}

With an Argo workflow trigger, we will templatize creating a Workflow CRD to take the SQS queue, which we defined in our dependencies, and define it as a parameter. The body data key will be abstracted for later use as the value of the first input parameter.

...
# Continuation under spec
  triggers:
    - template:
        name: sqs-workflow
        k8s:
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: aws-sqs-workflow-
                namespace: argocd
              spec:
              ...
          parameters:
            - src:
                dependencyName: test-dep
                dataKey: body
              dest: spec.arguments.parameters.0.value

Within the spec, we key that parameter with the name of the message, then templatize that key input parameter as a command line argument for our container service.

# Continuation under spec.triggers.0.template.spec
                entrypoint: main
                arguments:
                  parameters:
                  - name: message
                    value: overridden
                templates:
                - name: main
                  inputs:
                    parameters:
                    - name: message
                  container:
                    image: docker.io/org/container:latest
                    args: 
                      - "-m"
                      - "{{inputs.parameters.message}}"

End to End Process

When a message gets submitted to the SQS queue, the event source pod will pick it up and will trigger the Sensor. Once triggered, an Argo Workflow CRD is created where end to end, a message is delivered from SQS and processed inside your cluster.

Summary

The integration of Argo Events and Workflows connects FaaS to Kubernetes, providing numerous benefits through this orchestration duo.The power of Sensors as intermediaries, that links external systems and Kubernetes environments, offers a simple pathway to exchange data and actions. This connectivity is exemplified through integration with AWS Simple Queue Service (SQS), which enhances adaptability and seamlessly scales systems.

Terraform Workspaces for Local AWS Development

Daniel Quackenbush — Mon, 14 Mar 2022 00:38:59 +0000

There are times when you need to develop the AWS API but could get away with merely imitating it. Using containers, we can stand up emulated AWS services. This can help when say… you’re completely disconnected or just want to save money while developing. But how do you continue with your typical IaC workflow? Terraform allows you to override the AWS API URLs by explicitly setting local endpoints. In this post, I’ll show you how to use containers and Terraform workspaces to develop the AWS API 100% locally!

Define AWS Provider

We will initialize the provider block with our region configuration in our terraform directory. Then with the use of dynamic blocks, we can set individual endpoints. In this case, we want to configure the DynamoDB endpoint. If using localstack, you can expand this to a larger array of services.

provider "aws" {
  region = "us-east-2"

  dynamic "endpoints" {
    for_each = terraform.workspace == "local" ? [1] : []
    content {
      dynamodb = "http://localhost:8000"
    }
  }
}

Create Dynamo Resource

resource "aws_dynamodb_table" "workspaces" {
  name           = "workspaces"
  billing_mode   = "PROVISIONED"
  read_capacity  = 1
  write_capacity = 1
  hash_key       = "id"

  attribute {
    name = "id"
    type = "S"
  }
}

Start dynamo docker container

docker run --rm -d -p 8000:8000 amazon/dynamodb-local

Create and Select Workspace

If this is the first time, you will want to create the local workspace:

terraform workspace new local

Otherwise, if switching between local and the default workspace

terraform workspace select local

Run Terraform Apply

~ terraform apply                
...
aws_dynamodb_table.workspaces: Creating...
aws_dynamodb_table.workspaces: Creation complete after 0s [id=workspaces]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now we can validate the table exists locally and not in AWS

### Remote
~  aws dynamodb list-tables
{
    "TableNames": []
}
### Local
~ aws dynamodb list-tables --endpoint http://localhost:8000
{
    "TableNames": [
        "workspaces"
    ]
}

Provision in AWS

To provision the resource now in AWS you can switch back to the default namespace, and run terraform apply again.

~ terraform workspace select default

~ terraform apply -auto-approve                                   
...
aws_dynamodb_table.workspaces: Creating...
aws_dynamodb_table.workspaces: Still creating... [10s elapsed]
aws_dynamodb_table.workspaces: Creation complete after 13s [id=workspaces]

~ aws dynamodb list-tables
{
    "TableNames": [
        "workspaces"
    ]
}

Configuring AWS SSO

Daniel Quackenbush — Fri, 18 Feb 2022 02:31:41 +0000

Terraform provides several resources for configuring AWS SSO across an organization. Once the service is enabled, you will need to define an identity source. This can be using the built-in directory service, active directory, or any external identity provider with SAML integration. At this time of writing, identitystore doesn't have a fully fleshed out API, so you will have to configure this manually.

However, once the identity store is configured, it can utilize those pushed or self-created users and groups and assign permission sets to accounts.

Get SSO Instance ID and Identity Group Via Lookup

data "aws_ssoadmin_instances" "this" {}
data "aws_identitystore_group" "this" {
  identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]

  filter {
    attribute_path  = "DisplayName"
    attribute_value = var.group_name # Fill in the group you defined
  }
}

Create a Permission Set to Define Accounts

resource "aws_ssoadmin_permission_set" "this" {
  name             = var.policy_name
  description      = var.policy_description
  session_duration = "PT12H" # Set this duration to the time you desire
  instance_arn     = tolist(data.aws_ssoadmin_instances.this.arns)[0]
}

Define Policy For Permission Set

Managed Policy

If you have a list of managed polcies you'd like to attach, you can loop over and attach them indiviudally.

resource "aws_ssoadmin_managed_policy_attachment" "this" {
  for_each           = toset(var.managed_policy_arn)
  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  managed_policy_arn = each.value
  permission_set_arn = aws_ssoadmin_permission_set.this.arn
}

Inline Policy

data "aws_iam_policy_document" "sample_bucket_read" {
  statement {
    sid = "0"
    actions = [
      "s3:GetObject"
    ]
    resources = [
      "arn:aws:s3:::sample-bucket/*"
    ]
  }
}

resource "aws_ssoadmin_permission_set_inline_policy" "this" {
  inline_policy      = data.aws_iam_policy_document.sample_bucket_read.json
  instance_arn       = aws_ssoadmin_permission_set.this.instance_arn
  permission_set_arn = aws_ssoadmin_permission_set.this.arn
}

Apply the permissions sets to Accounts

data "aws_organizations_organization" "this" {}

resource "aws_ssoadmin_account_assignment" "this" {
  for_each           = toset(data.aws_organizations_organization.this.accounts[*].id)
  instance_arn       = aws_ssoadmin_permission_set.this.instance_arn
  permission_set_arn = aws_ssoadmin_permission_set.this.arn
  principal_id       = data.aws_identitystore_group.this.group_id
  principal_type     = "GROUP"
  target_id          = sensitive(each.value)
  target_type        = "AWS_ACCOUNT"
}

Mocking the AWS SDK With Go

Daniel Quackenbush — Wed, 08 Dec 2021 17:22:35 +0000

Mocking a client library is a common technique when building test-driven development. In golang, this can be done by creating structs that implement interfaces and then override the methods you are trying to mock. This example of mocking can be done with any method, but for this post, I will use AWS Organizations to demonstrate.

Implementation Code

First, we want to create a struct that will be used as the methods' instance. As you see below, we are going to implement the interface of the SDK provided organizations API:

type Organizations struct {
    Client organizationsiface.OrganizationsAPI
}

Then once we have defined that, we will have to instantiate the method we want to implement, which for the main implementation will be a pass-through for the method used by the client.

func (s *Organizations) ListAccounts(in *organizations.ListAccountsInput) (*organizations.ListAccountsOutput, error) {
    result, err := s.Client.ListAccounts(in)
    return result, err
}

Finally, we will want to create a method we will test. We must parameterize the struct to help our test define which method to use.

func WhatAreMyAccounts(client *Organizations) (*organizations.ListAccountsOutput, error) {
    return client.ListAccounts(
        &organizations.ListAccountsInput{
            MaxResults: aws.Int64(5),
            NextToken:  nil,
        },
    )
}

Test Code

For testing, we will create our Mock struct, followed by our method override, and then wrap that within a given test. The MockOrganization struct will implement an interface, which we can later utilize as the organization's client.

type MockedOrganizations struct {
    organizationsiface.OrganizationsAPI
}

For the test, we want to guarantee a response. Therefore, we will define a separate implementation of the organization interface, of which we will return a constant value for the test suite.

func (m *MockedOrganizations) ListAccounts(in *organizations.ListAccountsInput) (*organizations.ListAccountsOutput, error) {
    return &organizations.ListAccountsOutput{
        Accounts: []*organizations.Account{
            {
                Arn:   aws.String(""),
                Email: aws.String("test1@example.com"),
                Id:    aws.String("234567890"),
                Name:  aws.String("test-1"),
            },
            {
                Arn:   aws.String(""),
                Email: aws.String("test2@example.com"),
                Id:    aws.String("123456789"),
                Name:  aws.String("test-2"),
            },
        },
    }, nil
}

On the test code, by using our mocked interface, we can create an Organizations object. The application's function then calls our mocked response to yield the result to test.

func TestListAccounts(t *testing.T) {
    test := Organizations{
        Client: &MockedOrganizations{},
    }
    resp, err := WhatAreMyAccounts(&test)
    assert.Equal(t, len(resp.Accounts), 2)
    assert.NoError(t, err)
}

Results

➜  main go test -v
=== RUN   TestListAccounts
--- PASS: TestListAccounts (0.00s)
PASS
ok      main/cmd/main   0.117s

To see the full code, check out the below gists:

Shift-left Infrastructure Security

Daniel Quackenbush — Thu, 17 Dec 2020 00:44:06 +0000

Bridging the gap between security and engineering can bring significant value in compliance and operational protection, and its impact will place broad strokes in knowledge transfer and relationship building. In functional shops, security might oversee rules, time-based audits, and create tickets for teams. As teams cross over the DevSecOps divide, security becomes less of a task and more verification in the release. Specifically focused on cloud workloads, several automated steps can be adopted that plug right in and give that instant feedback engineers desire. Call the movement: continuous security.

Infrastructure Pipelines

There are several moving parts with cloud infrastructure, regardless of which provider, albeit a data store, server, security group, or any other resource configuration. Several compliance institutions have come out with benchmarks to compare against, and these are what security uses in auditing. With shift left, the question is, how can we integrate with the workflows that already exist versus devising new solutions or performing out of band scanning? Several open-source tools can provide this protection.

Disclaimer: This piece is not designed to focus on tools, but rather how tools can simplify the process. No means suggested is a catch-all and should be seen as examples rather than implementation specifications.

Static Analysis of IAC

Depending on your stack, several solutions can provide coverage. Static analysis of infrastructure should be seen as different than static analysis of application code, as IAC is finite and declarative, where applications are variable. The tools in this market vary by the framework in which you are utilizing. TFSec, for example, is focused solely on terraform, whereas Checkov covers multiple frameworks. In determining what works best, the most critical factors to look for are extensibility and coverage.

Take this sample Terraform code, which creates an s3 bucket:

Utilizing a framework like checkov, we can run a series of checks against the resource definition and determine that some work can be done to harden the overall configuration.

With that said, checkov is not the only static code analysis. For a more comprehensive breakdown of static analysis tools, I recommend you read this blog post by Christophe Tafani-Dereeper.

Config Test

If we take a quote from Jim Brikman from Hashiconf 2018, "Infrastructure code without automated tests is broken." Inspec is widely adopted to ensure that the config is appropriately set, but what about using it for security? Several resource checks can validate encryption settings, access policies, etc. Take that same s3 bucket from before, and after deployment, we can validate to ensure the deployment meets our expectations.

~ inspec exec -t aws:// .

Profile: Bucket Testing profile (bucket-test)
Version: 0.1.0
Target:  aws://

  S3 Bucket tf-test-bucket
     ✔  is expected to exist
     ✔  is expected not to be public
     ×  is expected to have default encryption enabled
     expected #has_default_encryption_enabled? to return true, got false
     ✔

As you get writing more tests, there can become an issue when you have trickling dependencies. An example would be if you create an access policy, having the means to exercise that policy to validate that it is doing what it should, and not doing what it should not. Integration tests are where Terratest or self-curated scripts (ex. pytest + kubetest) are the means to the solution. An integration test will ensure that the written access policy will continue to work regardless of the underlying changes.

What if we had a role that we wanted to ensure could list objects from our s3 bucket? Can we deploy the role, then assume it and test its permissions against the bucket

~ go test
Running command terraform with args [init -upgrade=false]
...
Running command terraform with args [apply -input=false -auto-approve -lock=false]
...
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
...
PASS

Runtime protection

In this domain, we typically start to see tight integrations. In the k8s world, an example involves OPA Gatekeeper utilizing the admission controller; in the server space, the remediation might be a service such as AWS Config, which will scan existing assets against known configuration checks.

Recently a CVE was released within k8s, CVE-2020-8554. Using OPA Gatekeepr, you can create a constraint template, that restricts the ability from publishing through an admission controller

After trying to exercise the vulnerability, gatekeeper will block it.

Error from server ([denied by k8sexternalip] service has forbidden external IPs: {"23.185.0.3"}): error when creating "STDIN": admission webhook "validation.gatekeeper.sh" denied the request: [denied by k8sexternalip] service has forbidden external IPs: {"23.185.0.3"}

Application Pipelines

Many applications depend on system resources or external dependencies, of which the OWASP Top 10 calls out “Using Components with Known Vulnerabilities.” The nice thing about being in the OWASP Top 10, is engineers tend to be more aware of the risks. When talking about shifting left, there are several measures to keep you protected.

Utilize SCA to discover in code

Application dependencies are defined in a static config file, and most languages have a typical pattern for analysis. GitHub and GitLab, two leading Source Code Management solutions, have SCA built into their platform. However, security tools such as Synk or Contrast can integrate directly into your already existing application security stack. In the end, it’s not about which software, but rather ensuring your dependencies are up to date, and having an automated means to do so that runs against your existing CI pipeline makes it easy for developers to adopt.

A web app I manage utilizes AWS Amplify. Having an SCA tool at the disposal can keep the application up to date whenever a patch, minor, or major release comes out. Github's Dependabot compared Amplify's current package and saw a new one was released in this example. Auto opening a PR, the CI checks were able to execute and assure that it won't break the build.

Image Scanning

New CVEs are discovered daily. Having build-time and proactive scans against your images can help you find out OS-based patches that need to be applied. Scanning can take part both on the host side or the container side. Clair is a popular application integrated by many vendors for containers, but Nessus lead the way on the host side.

The claire scanner makes it easy to scan your containers on the fly. Once everything is all set up, you can scan any container image to get a list of its vulnerabilities.

Summary

Engineers feed into automation and want to utilize existing pipelines and frameworks to bridge the technical and social gap between security and engineering. Using tickets as a means of patching leads to slow turnarounds, and time-based audits lead to large openings of vulnerabilities. Through integrating changes on the infrastructure and application pipelines, the shift-left approach leads to an overall better security posture.

The Big Picture with Security Hub

Daniel Quackenbush — Wed, 16 Dec 2020 19:24:42 +0000

AWS Security Hub is a comprehensive analysis utility that gives a clearer picture of your infrastructure. Through native and vendor integrations, it provides a single lens on security while providing you actionable insight.

Security Hub utilizes foundational standards, CIS benchmarks, and PCI compliance checks to validate your infrastructure security. While using standards gets over the hurdle of managing rules, it still requires the overhead of enabling config and reporting back to the management account. However, despite the downsides, it can be a powerful utility in improving your security posture.

Implementation

All of the resources are needed, as described in AWS Config. In replacement of config rules, we use the conformance packs, aws-foundational-security-best-practices, and cis-aws-foundations-benchmark. The conformance packs then translate individual benchmarks into rules. It is worth noting that the checks' overall cost adds up quickly with more regions + accounts.

Findings

Security Hub provides an aggregated console that breaks down the findings' severity, a key to prioritizing work. An example of what the prioritization looks like:

Through navigating through the UI, you can then drill down to the specific AWS Config finding and view the non-compliant resources

Scores

Scores are also an interesting metric to pull out. When you look at the CIS benchmarks, it is nice to report back a score of improvements to track progress.

Integrations

Security Hub integrates with many services, not just AWS Config. Both AWS services and vendor services, overall giving you a regionalized view of your infrastructure state.

Governance with AWS Config

Daniel Quackenbush — Wed, 16 Dec 2020 19:14:14 +0000

AWS Config allows you to assess configuration based on predefined rules or custom rules based on lambda functions. The service can analyze rule sets such as encrypted RDS, public access denial, encrypted volumes, etc. These results can then be aggregated to a parent account through a two-way authorization handshake, providing one view for all benchmarks.

Implementation

In testing, I took a sandbox account and put the service to work.

Components Needed:

config configuration recorder
- the mechanism for snapshotting AWS resources
config delivery channel
- the means to deliver data from the recorder to the topic/bucket
S3 bucket
- the place where data is stored
SNS topic
config rule
- rules to analyze

Analysis:

Utilizing aggregation, it becomes seamless to view all the non-compliant resources to act on:

You can then drill into the respective resource to view the non-compliant resources and determine when the resource compliance changed.

Downsides

AWS Config can remove a lot of toil for compliance but can also be troublesome in some cases. Some of the limitations I discovered in researching:

Management overhead
- Code deployments of config rules, bucket, topic, etc. to every account. You must deploy all resources to all accounts; the aggregation feature does not permit the ruleset to be defined in one location.
Manual Config
- AWS provides a large number of rules, but all require manual enablement. If utilizing a vendor, they write the rules for you; moreover, you would become a rule manager.

Example Code

This code only has defined a single delivery channel in the provider region, us-east-1. If implemented, multiple delivery channels should be configured for multi-region support.

danquack / aws-config

Reference:

https://docs.aws.amazon.com/config/latest/developerguide/managed-rules-by-aws-config.html
https://docs.aws.amazon.com/config/latest/developerguide/aggregate-data.html