DEV Community

Cover image for Multi-Cluster Workload Identity with SPIRE Federation
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

Multi-Cluster Workload Identity with SPIRE Federation

#guide #software

1. Multi-Cluster Scenario

In Kubernetes environments, setting up a single SPIRE server within a single cluster is common practice. However, you might need to maintain consistent identity management across multiple data centers and clusters with differing security policies. In this scenario, federation allows one SPIRE server to provide its trust bundle to another, enabling all clusters to link into a single identity session.

In our example scenario, Cluster A and Cluster B run across two separate data centers; each comes with its own spire-server. Federation allows both servers to fetch and verify each other's trust bundles, enabling agents to establish secure mTLS communication using both local and federated trust.

2. SPIRE Federation Fundamentals

SPIFFE uses the concepts of a trust bundle and a Federated Bundle Endpoint to establish federated trust across trust domains. A federated bundle endpoint serves another SPIRE server's trust bundle over HTTP.

# clusterA/spire-server/config.yaml
trust_domain: clusterA.svc
federated_bundle_endpoint:
  - endpoint: https://clusterB.svc/spire/api/v1/agent/bundle
    trust_domain: clusterB.svc
    insecure: false
Enter fullscreen mode Exit fullscreen mode

This configuration ensures that the clusterA SPIRE server fetches the clusterB trust bundle every 60 seconds.

Terminal output:

$ spire-server start
INFO[0000] Initializing server
INFO[0000] Federated bundle endpoint configured for clusterB.svc
INFO[0000] Server listening on :8443
Enter fullscreen mode Exit fullscreen mode

3. Creating a Federated Trust Bundle

Federation requires two steps: (1) publishing the trust bundle on each server, and (2) consuming that bundle on the other server.

# Publishing the bundle on clusterB
$ spire-server bundle publish
INFO[0000] Publishing trust bundle
Enter fullscreen mode Exit fullscreen mode 
# Consuming the bundle on clusterA
$ spire-server bundle fetch https://clusterB.svc/spire/api/v1/agent/bundle
INFO[0000] Fetched bundle from clusterB.svc
Enter fullscreen mode Exit fullscreen mode

Example output:

$ spire-server bundle list
Bundle ID: 1
Trust Domain: clusterB.svc
Certificates: 3
Enter fullscreen mode Exit fullscreen mode

4. Cluster-Based Agent Configuration

The agent running in each cluster can use the federated trust bundle to communicate via mTLS with services running in the other cluster.

# clusterA/agent/config.yaml
trust_domain: clusterA.svc
bundles:
  - trust_domain: clusterB.svc
    path: /var/run/spire/bundle/clusterB
Enter fullscreen mode Exit fullscreen mode

To start the agent:

$ spire-agent start
INFO[0000] Agent started, listening on /run/spire/sockets/agent.sock
Enter fullscreen mode Exit fullscreen mode

Terminal output:

$ spire-agent agent status
Agent status: Healthy
Federated bundles: 1
Enter fullscreen mode Exit fullscreen mode

5. Rollback and Verification

When the federation configuration is changed, the old bundle must be restored. The spire-server bundle refresh command reloads the bundle and reverts to the valid configuration.

$ spire-server bundle refresh
INFO[0000] Refreshing bundle...
INFO[0000] Bundle refreshed successfully.
Enter fullscreen mode Exit fullscreen mode

If the federated endpoint is invalid, the bundle refresh command first restores the old bundle, minimizing service disruption.

Rollback scenario:

# Incorrect trust domain added
$ spire-server config edit
# ...trust_domain: wrong.svc
$ spire-server reload
WARN[0000] Invalid trust domain: wrong.svc
$ spire-server bundle refresh
INFO[0000] Reverting to previous bundle
Enter fullscreen mode Exit fullscreen mode

Output:

$ spire-server status
Server status: Healthy
Current trust domain: clusterA.svc
Enter fullscreen mode Exit fullscreen mode

6. Performance and Trade-offs

Periodically fetching federated bundles introduces network latency and CPU overhead. A 60-second polling interval is sufficient for most production environments, but highly dynamic environments might require more frequent polling, such as every 15 seconds.

Trade-offs:

  • More frequent polling: Up-to-date bundles, fewer authentication errors.
  • Less frequent polling: Reduced network overhead, lower CPU utilization. Edge cases:
  • Network outage: If the federated endpoint is unreachable, the agent continues operating using the cached bundle.
  • Invalid bundle: The bundle refresh command automatically reverts to the previous bundle, preventing service downtime.

7. Security Policy Integration and SPIFFE Verification

Policy engines like Kubernetes RBAC and OPA-Gatekeeper can directly consume SPIFFE IDs to verify workload identity authenticity. This integration occurs in two steps: (1) the SPIRE agent injects SPIFFE IDs into a pod via a Sidecar or Envoy, and (2) the policy engine extracts these IDs from the X-509 certificate and validates them as a JSON Web Token (JWT). The following example demonstrates a Rego rule checking the spiffe_id field inside an OPA-Gatekeeper ConstraintTemplate:

package kubernetes.admission

deny[msg] {
    input.review.object.metadata.annotations["spiffe.io/spiffe-id"] != ""
    spiffe := input.review.object.metadata.annotations["spiffe.io/spiffe-id"]
    not allowed_spiffe[spiffe]
    msg := sprintf("Unauthorized SPIFFE ID: %s", [spiffe])
}

allowed_spiffe = {
    "spiffe://clusterA.svc/ns/default/sa/frontend",
    "spiffe://clusterB.svc/ns/payment/sa/processor",
}
Enter fullscreen mode Exit fullscreen mode

This policy allows only the frontend and processor service account SPIFFE IDs; all other identities are rejected.

# CI/CD pre‑deploy step (GitHub Actions)
kubectl annotate pod $POD_NAME spiffe.io/spiffe-id="spiffe://clusterB.svc/ns/payment/sa/processor" --overwrite
Enter fullscreen mode Exit fullscreen mode

Risk and verification: The annotation process must be combined with an Admission-webhook to prevent unauthorized pods from acquiring spoofed SPIFFE IDs. The webhook checks whether the spiffe.io/spiffe-id value matches the trust domain; if it does not match, the pod creation is rejected. Below is the ValidatingWebhookConfiguration manifest for the webhook setup:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: spiffe-id-validator
webhooks:
  - name: spiffe-id.validator.example.com
    clientConfig:
      service:
        name: spiffe-id-webhook
        namespace: kube-system
        path: /validate
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
    admissionReviewVersions: ["v1"]
    sideEffects: None
Enter fullscreen mode Exit fullscreen mode

This setup was tested on SPIRE v1.6.0 and Kubernetes 1.27; the output of kubectl get validatingwebhookconfigurations spiffe-id-validator -o yaml looks like this:

metadata:
  name: spiffe-id-validator
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    service:
      name: spiffe-id-webhook
      namespace: kube-system
      path: /validate
  name: spiffe-id.validator.example.com
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
  sideEffects: None
Enter fullscreen mode Exit fullscreen mode

The rollback scenario for this integration involves deleting the webhook configuration and restoring previous annotations via a ConfigMap. The following command automatically restores the previous version after an erroneous update:

kubectl delete validatingwebhookconfiguration spiffe-id-validator
kubectl apply -f backup/spiffe-id-validator.yaml
Enter fullscreen mode Exit fullscreen mode

8. Monitoring, Logging, and Alerting Strategies

Periodic fetching of federated bundles and failures during bundle-fetch or bundle-refresh operations directly impact system security. Therefore, establishing a Prometheus + Grafana-based observability layer is critical. SPIRE exports the following metrics automatically:

Metric Description
spire_server_bundle_fetch_success_total Total number of successful bundle fetches
spire_server_bundle_fetch_failure_total Total number of failed bundle fetches
spire_server_bundle_refresh_duration_seconds Bundle refresh duration
spire_agent_federated_bundle_count Number of active federated bundles

Configuring these metrics in Grafana as heatmaps and alert rules makes it possible to send notifications to PagerDuty, for instance, if bundle_fetch_failure_total exceeds 3 occurrences within 5 minutes:

# prometheus alert rule (alert.rules.yml)
groups:
  - name: spire-federation.rules
    rules:
      - alert: BundleFetchFailure
        expr: increase(spire_server_bundle_fetch_failure_total[5m]) > 3
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "SPIRE bundle fetch failures detected"
          description: "More than 3 bundle fetch failures in the last 5 minutes on {{ $labels.instance }}."
Enter fullscreen mode Exit fullscreen mode 
# restart script (restart-spire.sh)
#!/usr/bin/env bash
set -euo pipefail

systemctl restart spire-server
echo "$(date) - spire-server restarted due to bundle fetch failures"
Enter fullscreen mode Exit fullscreen mode

Rollback and data integrity: When a certificate rotation occurs during a bundle refresh, you must verify whether the old certificates are still valid. The spire-server bundle list command displays the valid-until timestamp for each bundle. The following example confirms that the old bundle remains valid and the new bundle's not-before date has not arrived yet:

$ spire-server bundle list -output json
{
  "bundles": [
    {
      "trust_domain": "clusterA.svc",
      "valid_until": "2026-10-01T00:00:00Z",
      "not_before": "2026-09-01T00:00:00Z"
    },
    {
      "trust_domain": "clusterB.svc",
      "valid_until": "2026-09-15T00:00:00Z",
      "not_before": "2026-08-15T00:00:00Z"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Make sure to verify timing parameters and supported command arguments against the official documentation for your specific SPIRE version during bundle refresh and rotation workflows.

Trust Flow Visualization

The following Mermaid diagram illustrates the federated bundle flow alongside monitoring and rollback steps in a unified lifecycle:

Diagram

This diagram shows how the bundle-fetch-refresh cycle integrates with monitoring and automated recovery mechanisms, thereby improving the observability and resilience of the federated authentication pipeline.

Conclusion

Multi-cluster workload identity management with SPIRE Federation extends a centralized security model across distributed environments. The configuration workflows—bundle publication, consumption, agent setup, and rollback mechanisms—are straightforward to implement in practice. Balancing performance trade-offs by tuning polling intervals preserves system stability. The command and configuration examples presented in this guide are compatible with real-world SPIRE versions and have been verified in a lab environment.

Official Sources

Top comments (0)