DEV Community: Vipin Vijaykumar

SAP BTP Multi-Region Setup with Terraform: A Practical Guide to Disaster Recovery Subaccount Pairing

Vipin Vijaykumar — Sun, 19 Jul 2026 16:16:27 +0000

From real-world outages to automated resilience — how to protect your SAP BTP workloads across regions using Terraform

When the Cloud Goes Down

In March 2026, physical attacks on data center infrastructure in the Middle East took two of three availability zones in an AWS region completely offline. The damage was not a software bug or a misconfiguration — it was fire and power shutdown from direct physical impact. Engineers could not simply reboot their way out of it. Physical repairs took multiple days.

The cascading effect went further than the directly hit facilities. Ninety-one AWS services were disrupted across two Middle East regions. The control plane degraded — IAM, CloudFormation, and EC2 APIs all throttled — meaning that organizations whose DR strategy relied on deploying infrastructure during the incident found their recovery tooling was itself unreliable.

Analyses published after the incident landed on a blunt conclusion: Multi-AZ is not disaster recovery. One expert put it plainly: "Multi-AZ protects you from hardware failures, not a missile hitting an entire availability zone cluster in the same city."

A financial analysis of the impact illustrated the stakes concretely: for an $80K/hour SaaS platform, total costs from such an outage — revenue loss, SLA penalties, customer churn — would reach $2.77 million. Organizations that had pre-provisioned infrastructure in a secondary region and implemented automated, data-plane-based failover recovered in 28 minutes. Those that did not faced days of downtime.

Network World's post-incident report captured the lesson: "The enterprises that will come through events like this aren't the ones with the thickest DR binders, they're the ones who've actually failed over to another continent."

These are not theoretical scenarios. They happened.

What Is Disaster Recovery?

Disaster recovery (DR) is the practice of designing systems so that when a failure occurs — and failure always occurs eventually — your business can continue operating or recover fast enough that the impact is acceptable.

Two metrics define every DR strategy:

Recovery Time Objective (RTO) — the maximum acceptable time your system can be down after a failure. This is a business decision, not a technical one.

Recovery Point Objective (RPO) — the maximum acceptable data loss measured in time. Can you tolerate losing the last 15 minutes of transactions? The last 24 hours? Nothing at all?

These are defined in the NIST SP 800-34 Contingency Planning Guide and the AWS Well-Architected Reliability Pillar. The lower your RTO and RPO targets, the more investment your DR strategy requires — but according to the Uptime Institute's 2023 Annual Outage Analysis, more than two-thirds of enterprise outages already cost over $100,000. For large enterprises the figure climbs past $1 million per hour. The cost of not investing is often higher than the cost of investing.

Multi-AZ: Necessary but Not Sufficient

Before discussing multi-region, it is worth being precise about what multi-AZ actually protects against — because it is commonly misunderstood.

Cloud providers divide each geographic region into multiple Availability Zones (AZs) — physically separate data centers within the same metropolitan area, connected by low-latency private fiber. Deploying across multiple AZs within the same region protects against:

A single data center power failure
A single building fire or hardware fault
A localized network partition within one facility

What multi-AZ does not protect against:

A failure that affects the entire regional network layer (routing misconfiguration, backbone outage)
A physical event large enough to affect multiple facilities in the same city simultaneously
A regional control plane failure that takes down APIs across all AZs
Legal or regulatory action that makes an entire geographic jurisdiction inaccessible
A submarine cable cut that severs connectivity to the entire region

The March 2026 Middle East incident demonstrated all of these simultaneously. The AZs were in the same metropolitan area. When two of three were hit, the third experienced cascading degradation from the loss of redundancy. And when the control plane throttled, the customers whose DR plans said "redeploy in a secondary region" discovered they could not reliably execute that plan using the primary region's APIs.

A real-world case study of a multi-region DR implementation shows what proper architecture achieves: after a regional outage caused nearly a full day of downtime with $40,000 in losses, the organization moved to an active/passive architecture across two geographically separate regions. The result was a 2–5 minute RTO and sub-30-second RPO — at 50% of the cost of a full active-active model.

Multi-region is not a luxury for hyperscalers. It is the minimum viable resilience posture for business-critical workloads.

The Physical Internet Is Also a Factor

The Red Sea submarine cable incidents of early 2024 are a useful reminder that cloud regions are not abstract — they depend on physical cables laid on ocean floors. When four major cables connecting Europe, the Middle East, and Asia were damaged, approximately 25% of traffic on those routes was disrupted. Providers rerouted traffic around the Cape of Good Hope, adding latency and reducing throughput. Cloud services in European regions serving users in Asia experienced degraded performance for weeks while repairs were negotiated.

Organizations with active workloads on both sides of the disruption — in both an EU region and an Asia-Pacific region — were insulated. Those with all workloads in a single region were not.

The point is not that these events happen frequently. The point is that they happen unpredictably, they affect entire geographic areas, and multi-AZ provides no protection against them.

SAP BTP Multi-Region: The Platform-Level Response

SAP Business Technology Platform is the cloud foundation for modern SAP applications — the runtime for extensions, integrations, and services that connect to core SAP systems. For enterprises running business-critical SAP workloads on BTP, regional resilience is not optional.

SAP's answer is the BTP Multi-Region Setup, described in the SAP BTP Multi-Region Guide. It provides a native disaster recovery capability at the platform layer, built around the following core ideas:

Basic Concepts

SAP defines specific region pairs — the secondary region for a given primary is prescribed by SAP, ensuring sufficient geographic distance while keeping data within the same jurisdiction. The pairing is not arbitrary; it reflects deliberate decisions about latency, data residency, and regulatory compliance.

The BTP Multi-Region architecture involves:

Region Pairs — SAP-defined pairs of regions that are sufficiently distant from each other while remaining within the same country or jurisdiction
Subaccount Pairs — two customer subaccounts, one in each region of a pair, linked into a DR relationship
Redundant Setup — customers deploy equivalent resources in both subaccounts; SAP BTP handles asynchronous replication between them behind the scenes
Instance-level Failover — controlled failover for individual service instances
Subaccount-level Failover — controlled failover of an entire subaccount (currently in development)
Regional-level Failover — in a catastrophic regional event, SAP can initiate failover after a formal disaster declaration

One important design note from the SAP documentation: customers should segregate DR and non-DR workloads into separate subaccounts.

Currently Supported Region Pairs

SAP maintains a defined and growing set of DR region pairs. The secondary region for a given primary is prescribed by SAP ensuring sufficient geographic distance while keeping data within the same jurisdiction. For the most current list of supported pairs, refer to the SAP BTP Multi-Region Pairs page.

How to Set Up Multi-Region on BTP

According to the SAP setup guide, the process involves:

1. Pair two subaccounts — one in each region of a supported pair. The pairing is symmetric: SAP BTP does not maintain or enforce "primary" and "secondary" roles at the platform level. You define which is primary based on your use case, typically proximity to your users.

2. Configure redundant resources — customers are responsible for deploying equivalent resources in both subaccounts. This includes:

Trust configuration (same Identity Provider in both subaccounts for seamless authentication after failover)
Entitlements (equivalent service entitlements in both subaccounts)
Role collections and assignments
Destination and connectivity settings
Environment variables and service bindings

3. Pair service instances — for services with built-in multi-region support, create a paired instance using the dr_setup parameter:

cf create-service <service> <plan> <name> \
  -c '{"dr_setup": {"op": "pair", "values": ["<primary_instance_id>"]}}'

After pairing, the primary instance handles traffic while data is asynchronously replicated to the secondary.

For the full list of services with built-in multi-region support (a growing list), refer to SAP BTP Services with Built-in Multi-Region Support.

Triggering a Failover

When a failover is needed, SAP BTP supports two strategies:

Planned failover — zero data loss. The primary service instance becomes read-only, the secondary waits for full replication to complete, then promotes itself. If replication cannot complete within a time window, the failover is aborted. This is for scheduled maintenance or proactive region migration.

Forced failover — used during regional outages where the primary is unreachable. May incur data loss. The customer is responsible for resolving any data inconsistency afterward.

cf update-service <name> \
  -c '{"dr_setup": {"op": "promote", "values": ["forced"]}}'

Why Automate This with Terraform?

Subaccount pairing through the BTP cockpit works for one environment. In practice you have multiple subaccounts in development, staging, and production — each needing its own pairing, each with the same bilateral role assignments, each needing to be documented, audited, and reproducible.

The SAP BTP Terraform provider exposes subaccount pairing as a first-class resource. With it, your DR topology is:

Auditable — every pairing change is a git commit with an author and timestamp
Repeatable — new environment, same topology, terraform apply
Drift-detectable — terraform plan catches manual changes made through the cockpit
GitOps-compatible — DR changes go through pull request review like any other infra change

Critically: the March 2026 incident showed that relying on deploying infrastructure during an outage is dangerous — the control plane may be degraded. With Terraform, everything is pre-provisioned and pre-declared. When the incident happens, the infrastructure is already there.

The Terraform Resource: `btp_disaster_recovery_subaccount_pair`

Refer to the full schema from the Terraform Registry:

Required Arguments

Argument	Type	Description
`subaccount_id`	String	The ID of the first subaccount to pair
`paired_subaccount_id`	String	The ID of the second subaccount to pair

Read-Only Attributes

Attribute	Type	Description
`pair_id`	String	Unique identifier of the subaccount pair
`globalaccount_id`	String	The Global Account owning both subaccounts
`created_by`	String	The user who created the pairing
`created_date`	String	Timestamp of creation in RFC3339 format

Step-by-Step Terraform Configuration

1. Provider Setup

provider.tf

terraform {
  required_providers {
    btp = {
      source  = "SAP/btp"
    }
  }
}

provider "btp" {
  globalaccount = var.globalaccount_subdomain
}

2. Variables

variables.tf

variable "globalaccount_subdomain" {
  type        = string
  description = "Your BTP Global Account subdomain"
}

variable "primary_subaccount_id" {
  type        = string
  description = "ID of the subaccount in the primary region (e.g. cf-eu10)"
}

variable "secondary_subaccount_id" {
  type        = string
  description = "ID of the subaccount in the paired secondary region (e.g. cf-eu13)"
}

3. The DR Pair Resource

main.tf

resource "btp_disaster_recovery_subaccount_pair" "dr_pair" {
  subaccount_id        = var.primary_subaccount_id
  paired_subaccount_id = var.secondary_subaccount_id
}

Two arguments. The provider handles the API calls to establish the DR relationship. That is the entire configuration needed to pair the subaccounts.

4. Outputs

outputs.tf

output "dr_pair_id" {
  value       = btp_disaster_recovery_subaccount_pair.dr_pair.pair_id
  description = "The ID of the disaster recovery subaccount pair"
}

output "dr_pair_created_by" {
  value       = btp_disaster_recovery_subaccount_pair.dr_pair.created_by
}

output "dr_pair_created_date" {
  value       = btp_disaster_recovery_subaccount_pair.dr_pair.created_date
}

5. Apply

terraform init
terraform plan
terraform apply

After apply, the pairing appears in the BTP cockpit under Multi-Region Setup in both subaccounts.

Importing an Existing Pair

If the pairing was created manually through the cockpit, import it without destroying and recreating it.

CLI import:

terraform import btp_disaster_recovery_subaccount_pair.dr_pair \
  'dd005d8b-1fee-4e6b-b6ff-cb9a197b7fe0,2dc1ecf1-786c-4f92-91f2-26650ab3ad28'

Import block (Terraform 1.5+):

import {
  to = btp_disaster_recovery_subaccount_pair.dr_pair
  identity = {
    subaccount_id        = "dd005d8b-1fee-4e6b-b6ff-cb9a197b7fe0"
    paired_subaccount_id = "2dc1ecf1-786c-4f92-91f2-26650ab3ad28"
  }
}

Important Constraints

One-to-one only. Each subaccount can be paired with exactly one other subaccount.
Bilateral role requirement. The Central Disaster Recovery Administrator role must be assigned in both subaccounts. Assigning it only in one is the most common setup mistake.
Destroy unpairs, does not delete. terraform destroy removes the pairing relationship — subaccounts and their resources are untouched.
Subaccount pairing is the foundation. It establishes the trust relationship. Individual service instance pairs and subscription pairs must be created on top of it to achieve actual data replication.
Separate DR from non-DR workloads. SAP recommends keeping DR-enabled resources in dedicated subaccounts, separate from non-DR workloads.

Summary

The events of the past few years — data center disruptions, undersea cable damage, regional network failures — have made one thing clear: multi-AZ resilience is no longer enough for business-critical workloads. The failure modes that actually cause extended outages operate at a level above individual availability zones.

Multi-region disaster recovery is the architectural response. For SAP BTP workloads, the platform provides this natively through subaccount pairing and service-level replication. Terraform makes the configuration auditable, repeatable, and safe to execute before the incident happens — not during it.

The full setup is genuinely simple:

resource "btp_disaster_recovery_subaccount_pair" "dr_pair" {
  subaccount_id        = var.primary_subaccount_id
  paired_subaccount_id = var.secondary_subaccount_id
}

Step	Action
1	Create two subaccounts in a supported SAP BTP region pair
2	Assign `Central Disaster Recovery Administrator` in both
3	Configure the SAP BTP Terraform provider
4	Declare `btp_disaster_recovery_subaccount_pair` and apply
5	Mirror entitlements, trust config, roles, and destinations
6	Create service instance pairs for each critical service
7	Test a planned failover — before you need it

References

Bye-Bye Credentials! Automate BTP & Cloud Foundry Setup with Terraform using Github Actions and Github OIDC

Vipin Vijaykumar — Wed, 13 Aug 2025 15:38:36 +0000

Imagine this: You want to automate setting up SAP BTP Accounts and Cloud Foundry using Terraform, but without the pain of managing and tracking usernames, passwords, or rotating any of those credentials.

Well, that's no longer a dream! Most common CI/CD Solutions like Github/Gitlab actions, Jenkins, Azure DevOps etc and OIDC Provider integration, you can wave goodbye to maintaining long-lived credentials. Say hello to short-lived, secure, just-in-time tokens.

In this blog, we’ll set up GitHub’s OIDC provider to authenticate directly against SAP BTP using JWT Bearer Assertion flow, and then use Terraform to provision your BTP resources, Cloud Foundry spaces, and more. These steps work for both Terraform and OpenTofu.

What's a JWT Bearer Assertion Anyway?

Before we jump into Terraform and GitHub Actions, let’s talk JWT Bearer tokens. A JWT Bearer Assertion is a way for a client to prove its identity to an authorisation server using a signed JSON Web Token (JWT) instead of a password or traditional credentials. Since the tokens are short-lived and ephemeral, they bring somewhat better security and rotation.

In short: Instead of passing around long-term credentials, GitHub can request a short-lived token for your pipeline, signed and trusted via OpenID Connect. This token is accepted by an SAP Cloud Identity Service tenant, which then lets Terraform (on behalf of GitHub Actions) access BTP and Cloud Foundry APIs.

Event Sequence

Github Workflow is triggered
The Github OIDC Provider is called, and a JWT Token is generated for the workflow. This token is stored in environment variables or github variables
The JWT Bearer token is used by the BTP & CF terraform provider in the Terraform scripts
The calls for the respective providers are authenticated in BTP and CF (UAA) via the trusted SAP Cloud Identity Service tenant. The bearer token is forwarded to the SAP Cloud Identity Service
SAP Cloud Identity services validate the token against the Github OIDC Public Keys(JWKS)
If the key is successfully validated, the clients are authenticated, and subsequent calls can be made.

Let’s dig in!

This tutorial builds upon a community blog that showcases how the cf CLI can leverage GitHub OIDC for logging into and managing Cloud Foundry.

We'll reference the community blog for the initial setup steps as it's well-documented there.

Step 1: Establish Trust between SAP BTP & SAP Cloud Identity Services

Please refer to the official documentation for doing the same

Step 2: Create the Github Repo

Create the github repo with repository-name in the Organisation you need. This repository and name will be used in the next steps.

Step 3:Configure SAP Cloud Identity Services

In this step, configure the SAP Cloud Identity Services to

Add the Repo User for Github Action Workflows. Note:
- Name format: repo/<Organisation>/<repository-name>
- Email format: repo/<Organisation>/<repository-name>@<github-domain>
Add the Repo User to a group
Configure Github OIDC as a Corporate IdP
Map the repository name as NameId (Enriched Token Claims)
Configure Identity Federation.
Set up the BTP Application to use the GitHub IDP

Please refer to the community blog mentioned above for the detailed steps with regards to this.

Step 4: Add user to Global Account

Add the repo-user email, created in SAP Cloud Identity Services in BTP.

Since we will be working with creating a subaccount and managing resources at the Global Account level, assign the Global Account Administrator Role Collection.

Step 5: Add content to the Github Repository

The following will be the structure of the project and files.

<repository-root-directory>/
├── .github
│   └── workflows
│       └── deploy.yml
├── main.tf
├── provider.tf
├── terraform.tfvars
└── variables.tf

Here we create a Terraform configuration that uses the Terraform providers for SAP BTP & Cloud Foundry to:-
Add the configuration to initialise the provider.tf file

`main.tf`

The main.tf has the definition of the infrastructure that will be created. For this article, we take a simple scenario as:-

Create a Subaccount
Initialise a Cloud Foundry Runtime Environment in the subaccount
Create a space within Cloud Foundry

This would, of course vary depending on your setup and scenario. So we won't delve into this section much.

For details on the different resources & datasources present in the Terraform providers for SAP BTP & Cloud Foundry, please refer to the registry.

resource "btp_subaccount" "project" {
  name      = "sample project"
  subdomain = "sample.project"
  region    = lower(var.region)
}

resource "btp_subaccount_environment_instance" "cloudfoundry" {
  subaccount_id    = btp_subaccount.project.id
  name             = "sample-cf-instance"
  landscape_label  = "cf-ap10"
  environment_type = "cloudfoundry"
  service_name     = "cloudfoundry"
  plan_name        = "standard"
  parameters = jsonencode({
    instance_name = "sample-cf-instance"
  })
}

resource "cloudfoundry_space" "space" {
  name = "dev_space"
  org  = btp_subaccount_environment_instance.cloudfoundry.platform_id
}

`variables.tf` file

Here we declare the variables globalaccount and idp. You'd have more depending on your script.

variable "globalaccount" {
  type        = string
  description = "The subdomain of the SAP BTP global account."
}

variable "idp" {
  type        = string
  description = "Orgin key of Identity Provider"
  default     = null
}

Configure `provider.tf`

By convention, the provider.tf file usually represents the configuration to initialise the providers. Here we will configure the SAP BTP and Cloud Foundry providers.

terraform {
  required_providers {
    btp = {
      source  = "sap/btp"
      version = "~> 1.15.0"
    }
    cloudfoundry = {
      source  = "cloudfoundry/cloudfoundry"
      version = "~> 1.8.0"
    }
  }
}

provider "btp" {
  globalaccount = var.globalaccount
  idp           = var.idp
}
provider "cloudfoundry" {
  api_url = "https://api.cf.${var.region}.hana.ondemand.com"
  origin  = var.idp
}

As can be seen:-

btp provider is configured with the global account and the idp (Based on the trust configured in the above steps)
cloudfoundry is configured with the cf-api and the origin(uaa), which is the same trust configuration as created above
Since the credentials used here is the assertion token, we'll be using the environment variables BTP_ASSERTION and CF_ASSERTION_TOKEN.

Note:- CF API depends on the region where the subaccount is created. Please refer to the right API in the official document

Step 6: Add the Github Workflow

The Github workflow initialises Terraform, generates the token and stores it as Github Outputs. These tokens are then assigned as environment variables to the Terraform Providers. Thereafter, the Terraform executes the scripts just as any typical workflow.

name: Deploy
on:
  workflow_dispatch:
    inputs:
      region:
        description: 'BTP Subaccount Region'
        required: true
        type: string
      idp:
        description: 'Identity Provider'
        required: true
        type: string
      global_account_id:
        description: 'Global Account ID'
        required: true
        type: string
      tenantIssuerUri:
        description: 'Issuer URL'
        required: true
        type: string
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
        id-token: write # This is required for requesting the JWT
        contents: read
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          terraform_wrapper: false
          terraform_version: latest

      - name: Install OIDC Client from Core Package
        run: npm install @actions/core @actions/http-client

      - name: Get Id Token
        uses: actions/github-script@v7
        id: idtoken
        env:
          TENANT_URL: ${{inputs.tenantIssuerUri}}
        with:
          script: |
            const tenanturl = process.env.TENANT_URL;
            const coredemo = require('@actions/core')
            let id_token = await coredemo.getIDToken(tenanturl)
            coredemo.setOutput('githubJwt', id_token)

      - name: Apply Terraform configuration
        run: |
          export BTP_ASSERTION="${{ steps.idtoken.outputs.githubJwt }}"
          export CF_ASSERTION_TOKEN="${{ steps.idtoken.outputs.githubJwt }}"
          terraform apply -auto-approve -var "region=${{ inputs.region }}" -var "idp=${{ inputs.idp }}" -var "globalaccount=${{ inputs.global_account_id }}"

For this article, the workflow is triggered manually based on user inputs. It takes the following inputs:-

BTP Subaccount Region: Region where the Subaccount will be created
Identity Provider: The name of the identity provider (Origin Key (Cloud Foundry)) that was configured in the Trust Configuration in BTP from the above step.
Global Account ID: The Global Account Subdomain
Issuer URL: The URL of the SAP Cloud Identity Service Tenant. Github OIDC Provider issues tokens for this tenant as the audience.

The token is generated in the step Get Id Token. In this article, we use the core package for the GH Actions Toolkit. An alternative approach of using curl. This approach can be seen in the blog referred above.

Step 7: Trigger Workflow

Trigger the workflow from the Github Actions tab. Add the necessary details as seen below.

The Job should run successfully, whereby you can see that the Token is generated and used by the subsequent steps.

Conclusion

By federating GitHub Actions with SAP Cloud Identity and leveraging the JWT Bearer Assertion flow, you can automate provisioning of SAP BTP resources and Cloud Foundry environments without storing or rotating long-lived service credentials.

Some Best Practices

Use any supported remote, encrypted Terraform/OpenTofu backends like HCP, Vault, OpenBao, S3/GCS/Azure Blob, etc, with server-side encryption.
Work with least privilege & role collections: In BTP, assign the repo user (or group) only the role collections needed for the automation (For e.g Avoid Global Account Admin if the setup is for a specific subaccount).
Ensure GitHub Repo is protected and require reviewers for workflows that can create or alter infrastructure.
Test your workflow setup on dev/test environments first.

Setting up OTEL Collectors for mTLS

Vipin Vijaykumar — Mon, 17 Feb 2025 03:27:47 +0000

What's mTLS

mTLS (Mutual Transport Layer Security) is an enhanced version of TLS (Transport Layer Security) where the client and the server authenticate each other during the connection establishment.

A very Simple Flow for mTLS. (Note arrow direction)

+--------+ ---1--> +--------+
| Client | <--2--- | Server |
+--------+ <--3--> +--------+

1a) Client sends request to server along with its certificate.
1b) Server verifies the client certificate using client's public key.(Server has the public Key in its store/DB)
2a) Server sends response along with its certificate.
2b) Client verifies the server certificate with the server public key
3) After mutual authentication, encrypted communication between client and server.

Typically, the server and client certificates are signed by a CA. As the server can manage multiple clients, the server usually maintains the public key of the CA that is used to sign all clients.

How is this different from regular TLS?

Regular TLS is unidirectional. I.E Only the client verifies the certificates presented by the server. The server does not verify the client's certificates.

Using mTLS in OpenTelemetry(OTEL)

To use mTLS between 2 OTEL components, we need client and server certificates.

Scenario

2 OTEL Collectors. One as client (exporter) and the other as server (receiver). Metrics are sent/forwarded from the client to the server. Since this is a demo setup (or internal) we will be using Self Signed certificates.

+-------------+                       +--------------+
| Otel Client |-------GRPC----------->|  OTEL Server |
| (Exporter)  |                       |  (Receiver)  |
+-------------+                       +--------------+

We'll be running this inside a Kubernetes cluster in 2 different namespaces.

Prerequisites

A Kubernetes cluster
kubectl & helm
System with OpenSSL client installed

Demo

1) Generate certificates

a)Generate rootCA certificate.
Since we are using Self-signed certificates, we will create and use our own rootCA.

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:4096 -subj '/O=A Team/CN=root.com' -keyout rootCA.key -out rootCA.pem

b)Generate server certificate
We need to first create a certificate signing request (csr) for the server certificate: server.csr and the key server.key

openssl req -out server.csr -newkey rsa:4096 -nodes -keyout server.key -subj "/CN=test.com/O=PM" -config server.cnf -extensions req_ext

You will notice a server.cnf config file. This file consists of extensions for adding Subject Alternative Names (SAN), which are alternative host names that can identify the server.

[req]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = 'test.com'
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = 'test.com'
DNS.2   = 'test.server.svc.cluster.local'
DNS.3   = 'localhost'
DNS.4   = 'test'
# add more for other names

We will then use the server.csr to create a signed server certificate: server.pem. The signing is done by the rootCA certificate that was generated in step a.

openssl x509 -req -sha256 -days 365 -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -in server.csr -out server.pem -extensions req_ext -extfile server.cnf

Notice: We use the rootCA certificate and private key to sign the certificate.

b)Generate client certificate
Similarly, we create client certificates & key: client.pem| client.key. For the ease of this demo , we use the same rootCA to sign the client certificates.

As before we use a client.cnf also maintained for SAN:

[req]
default_bits = 4096
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[ dn ]
CN = 'localhost'
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = 'localhost'

Creating the csr and signing it with the rootCA:

openssl req -out client.csr -newkey rsa:4096 -nodes -keyout client.key -subj "/CN=localhost/O=client" -extensions req_ext -config client.cnf

openssl x509 -req -sha256 -days 365 -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -in client.csr -out client.pem -extensions req_ext -extfile client.cnf

1) Deploy on Kubernetes

Prerequisite: Please install the opentelemetry-operator in the cluster.

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts

helm install otel-operator open-telemetry/opentelemetry-operator --namespace otel-collector-operator --create-namespace --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" --set admissionWebhooks.certManager.enabled=false --set admissionWebhooks.autoGenerateCert.enabled=true

a)Create namespaces
Create the 2 namespaces

kubectl create ns server
kubectl create ns client

b)Create Kubernetes secrets.
The certificates that were generated in step 1 will be used to create the secrets. These secrets will be mounted into the OTEL pods in the next step.

k create secret generic server -n server  --from-file=tls.pem=server.pem --from-file=tls.key=server.key --from-file=ca.pem=rootCA.pem # server certificates

k create secret generic client -n client  --from-file=tls.pem=client.pem --from-file=tls.key=client.key --from-file=ca.pem=rootCA.pem # client certificates

c)Deploy the OTEL Receiver (server)
The OTEL receiver is the server that accepts the metrics. For our demo purpose the OTEL server will receive the metrics (Port 4317) and print it in the logs. The server is deployed in server namespace and the secrets are mounted as volume to be used by the otel-collector receiver.

We also create a service named test with port 4317. While this is optional for the demo, its a good practice. Additionally certain meshes..For E.g istio relies on the port name for the protocol. (grpc is the name of the port in the service here)

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: test
  namespace: server
  labels:
    app: otel-collector
spec:
  mode: deployment
  volumeMounts:
  - name: server
    mountPath: /etc/pki/ca-trust/source/server-ca
    readOnly: true
  volumes:
  - name: server
    secret:
      secretName: server
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            #endpoint: test.server.svc.cluster.local:4317
            tls:
              cert_file: /etc/pki/ca-trust/source/server-ca/tls.pem
              key_file: /etc/pki/ca-trust/source/server-ca/tls.key
              client_ca_file: /etc/pki/ca-trust/source/server-ca/ca.pem
    exporters:
      # NOTE: Prior to v0.86.0 use `logging` instead of `debug`.
      debug:
        verbosity: detailed
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [debug]
---
apiVersion: v1
kind: Service
metadata:
  name: test
  namespace: test
spec:
  selector:
    app.kubernetes.io/component: opentelemetry-collector
    app.kubernetes.io/instance: test.test
    app.kubernetes.io/name: test-collector
  type: ClusterIP
  ports:
  - name: grpc # important for istio!
    protocol: TCP
    port: 4317

Verify the otel-collector instance and service is created

kubectl get pods,svc -n server

b)Deploy the OTEL Exporter (client)
The OTEL Exporter is the client that generates (i.e source) the metrics. We use the hostmetrics receiver plugin. This will vary according to the needs.
The other receiver is deployed in the client namespace. The client certificates are mounted as volumeMounts just as before.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: client
  namespace: client
  labels:
    app: client
spec:
  mode: deployment
  volumeMounts:
  - name: client
    mountPath: /etc/pki/ca-trust/source/client-ca
    readOnly: true
  volumes:
  - name: client
    secret:
      secretName: client
  config: |
    receivers:
      # Data sources: metrics
      hostmetrics:
        scrapers:
          cpu:
          disk:
    processors:
      # Data sources: traces
      attributes:
        actions:
          - key: environment
            value: external-test-3
            action: insert
      # Data sources: traces, metrics, logs
    exporters:
      otlp:
        endpoint: test.server.svc.cluster.local:4317
        tls:
          ca_file: /etc/pki/ca-trust/source/client-ca/ca.pem
          cert_file: /etc/pki/ca-trust/source/client-ca/tls.pem
          key_file: /etc/pki/ca-trust/source/client-ca/tls.key
      debug: #just to print metrics in logs of the client
    service:
      pipelines:
        metrics:
          receivers: [hostmetrics]
          processors: [attributes]
          exporters: [otlp,debug]

Verify the otelcollector instance is created

kubectl get pods -n client

c) Verify the setup
Check the logs of the server otel (receiver). You should see logs of metrics generated by the client otel (exporter).

kubectl logs -f <otel-receiver-pod> -n server

Accessing Privileged Containers in Cloud Foundry: A Guide

Vipin Vijaykumar — Thu, 19 Dec 2024 09:28:25 +0000

Usually, your productive apps don’t and shouldn't have SSH access in Cloud Foundry. However, you might need to SSH into an application for debugging purposes. Fortunately, this is achievable by enabling the SSH permission for both the space and the app and be enabled by the operator, after which you can use the cf ssh command. This approach suffices for most scenarios.

But sometimes, this level of access isn’t enough. You might require privileged access to the application container for advanced troubleshooting or debugging. Why might you need this? That’s a question only you can answer, but this blog will show you how it’s done.

Note: This process can only be executed if you are an operator of the Cloud Foundry deployment and can access the Diego cells.

Steps to Gain Privileged Access

Retrieve the app-guid for the application

cf app <app-name> --guid

bosh -d cf vms
bosh -d cf ssh <diego-cell>  #(any Diego cell)

Use the cfdot utility to locate application details

In Diego terms, applications are referred to as “lrps” (long-running processes). Using the app-guid, identify the lrp instance_id and the diego_cell where the application or process is running.

cfdot actual-lrps | grep -E '<app-guid>' | jq -r '. | select(.index == 0)' # Index 0 is for the 0th instance of the application.

This command will provide the instance_guid and cell_id of the application container.

SSH into the respective Diego Cell From the cell_id identified in the previous step, SSH into the corresponding Diego Cell.

bosh -d cf ssh <diego_cell>

Attach to the Garden container You can attach it to the container using one of two methods:

Approach 1: Using runc

sudo /var/vcap/packages/runc/bin/runc --root /run/containerd/runc/garden/ exec <instance_guid> --tty /bin/sh

Approach 2: Using ctr
This approach is not recommended as the ctr command line is not actively maintained and the commands may vary over time.

sudo /var/vcap/packages/containerd/bin/ctr -a /var/vcap/sys/run/containerd/containerd.sock -n garden tasks exec --exec-id my-shell --tty <instance_guid> /bin/sh

After running either of these commands, you will have privileged access to the application container.

! A Word of Caution !

With great power comes great responsibility.
Privileged access should be used sparingly and responsibly if possible NEVER. Always assess the necessity of these steps, and avoid this process unless required.
Things can break here if you are NOT careful!

I've messed up something what do I do

Fear not. Mostly deleting the app instance can reset and set things back to usual.

How to get into an app container manually in Cloud Foundry with Garden-Containerd Backend

Vipin Vijaykumar — Mon, 18 Nov 2019 09:54:26 +0000

Summary

A method to get into a container on the Diego Cell

Purpose

The requirement is to be able to access the application container in Cloud Foundry in scenarios where cf ssh would not be possible. Main reasons being troubleshooting inside the container. This write-up describes the procedures for the same. The original article for containers based on garden-runc is provided below.

Procedure

Obtain the app process-guid

cf curl /v2/apps/$(cf app <app-name> --guid)| jq -r '.metadata.guid + "-" + .entity.version'

Obtain the hosts in which the app is running. Choose one of the I

cf curl v2/apps/$(cf app <app-name> --guid)/stats | grep host

Login to the bosh director.
Identify the cell/VM Instance name associated to the IP
SSH into the VM

bosh -e <environment> -d <deployment> ssh <Instance-Name>
diego-cell/8cbe944d-d0cc-4158-802e-59426506faf7:~$ sudo -i
diego-cell/8cbe944d-d0cc-4158-802e-59426506faf7:~#

Obtain the "instance_guid" for the process-guid from step 1

cfdot actual-lrp-groups | grep d7c18647-9560-4327-8bcc-b7ab1ecdf295-2d6ebf81-7857-46d8-8bdc-f958795abf86

Attach into the container using ctr (containerd client)

export containerid=<Instance_Guid_>
/var/vcap/packages/containerd/bin/ctr -a /var/vcap/sys/run/containerd/containerd.sock -n garden tasks exec --exec-id my-shell --tty $containerd /bin/bash
root@9b5900f2-34f3-4317-63bb-863c:/#

You will now be in the container.

DEV Community: Vipin Vijaykumar

SAP BTP Multi-Region Setup with Terraform: A Practical Guide to Disaster Recovery Subaccount Pairing

When the Cloud Goes Down

What Is Disaster Recovery?

Multi-AZ: Necessary but Not Sufficient

The Physical Internet Is Also a Factor

SAP BTP Multi-Region: The Platform-Level Response

Basic Concepts

Currently Supported Region Pairs

How to Set Up Multi-Region on BTP

Triggering a Failover

Why Automate This with Terraform?

The Terraform Resource: btp_disaster_recovery_subaccount_pair

Required Arguments

Read-Only Attributes

Step-by-Step Terraform Configuration

1. Provider Setup

2. Variables

3. The DR Pair Resource

4. Outputs

5. Apply

Importing an Existing Pair

Important Constraints

Summary

References

Bye-Bye Credentials! Automate BTP & Cloud Foundry Setup with Terraform using Github Actions and Github OIDC

Let’s dig in!

Step 1: Establish Trust between SAP BTP & SAP Cloud Identity Services

Step 2: Create the Github Repo

Step 3:Configure SAP Cloud Identity Services

Step 4: Add user to Global Account

Step 5: Add content to the Github Repository

main.tf

variables.tf file

Configure provider.tf

Step 6: Add the Github Workflow

Step 7: Trigger Workflow

Conclusion

Some Best Practices

Setting up OTEL Collectors for mTLS

What's mTLS

How is this different from regular TLS?

Using mTLS in OpenTelemetry(OTEL)

Scenario

Demo

1) Generate certificates

1) Deploy on Kubernetes

Accessing Privileged Containers in Cloud Foundry: A Guide

How to get into an app container manually in Cloud Foundry with Garden-Containerd Backend

Summary

Purpose

Procedure

The Terraform Resource: `btp_disaster_recovery_subaccount_pair`

`main.tf`

`variables.tf` file

Configure `provider.tf`