DEV Community: Sri Sai Venkata Kasi Subbarayudu Kompella

Signing Container Images with Cosign

Sri Sai Venkata Kasi Subbarayudu Kompella — Mon, 08 Jun 2026 14:50:09 +0000

When you pull a container image from a registry, how do you know it actually came from a trusted source? How do you know no one tampered with it between the time it was built and the time it landed on your node?

This article walks through how image signing works, from the basics of supply chain security to hands-on cosign usage, and deep into the internals of key-based, keyless signing and GitHub Actions integration.

The Supply Chain Problem

When customers use your container image, they're placing a lot of trust in you. They're trusting that the software components included in the image are less vulnerable, that no one injected anything malicious, and that the image they're running is the exact artifact your pipeline built.

Supply chain security is the practice of giving customers confidence in the integrity and provenance of your software. Integrity means the artifact hasn't been modified. Provenance means you can prove where it came from and how it was built.

Two concepts we should be aware of:

SBOM (Software Bill of Materials):

A manifest of every software component that went into building the artifact: packages, versions, licenses. You can generate one using Trivy:

trivy image --format spdx-json kasisubbarayudu/buildkittest

This produces a JSON file that lists every package and its version in your image. If a CVE is dropped for a specific version of a software component, you can immediately know which of your images are affected.

Provenance:

It is the metadata describing how the artifact was built, when, and by which system. Together with an SBOM, provenance lets you trace an artifact back to its exact build pipeline run.

Generating this information is one thing. Making it tamper-proof and cryptographically verifiable is another. That's where signing comes in.

Why Sign Images?

Image signing directly addresses three supply chain risks:

Provenance: prove that an image was produced by your CI/CD system, not an attacker who got write access to your registry.
Integrity: detect if the image was modified after it was signed (any change to the image changes its digest, which would invalidate the signature)
Policy enforcement: Use tools like Kyverno/OPA Gatekeeper/Sigstore Policy Controller to block unsigned or unverified images from being deployed to your cluster at all.

In this article, we will use cosign to sign the images.

Key-Based Signing:

In case of key-based signing, we generate a key pair to sign the image and manage the key pair ourselves.

Generating a Keypair:

cosign generate-key-pair

This generates:

cosign.key: your private key, encrypted with a password you choose
cosign.pub: your public key, which you distribute freely to anyone who needs to verify your images

Signing an Image:

# Build and push your image first
docker build -t myregistry.io/myapp:v1.0 .
docker push myregistry.io/myapp:v1.0

# Sign it (will prompt for private key password)
cosign sign --key cosign.key myregistry.io/myapp:v1.0

One detail worth noting: cosign doesn't modify the image itself. The signature is stored as a separate artifact in the same registry, at a tag derived from the image digest:

myregistry.io/myapp:sha256-<digest>.sig

The original image is untouched, and the signature lives alongside it in the registry without requiring any special infrastructure.

Verifying a Signature:

cosign verify --key cosign.pub myregistry.io/myapp:v1.0

On success, cosign confirms the signature is valid and matches the public key. If the image has been tampered with or was never signed:

cosign verify --key cosign.pub myregistry.io/malicious:v1.0
# Error: no matching signatures

The Problem with Key-Based Signing:

Key-based signing works, but it comes with an operational burden: you're now running your own PKI. You need to manage the private key securely, rotate it, distribute the public key, handle revocation if the private key is compromised, and ensure the key isn't lost. For small teams, this can be manageable, but at scale, it becomes another piece of infrastructure to worry about.
This is the problem keyless signing was designed to solve.

Keyless Signing with Sigstore:

Keyless signing lets you sign artifacts without ever managing a long-lived private key. Instead of you holding the key, the signing key is ephemeral i.e it exists only for the duration of the signing operation and is then discarded. Identity is proven through OIDC.

The Sigstore ecosystem has three components that make this work:

Fulcio: a certificate authority that issues short-lived signing certificates
Rekor: an immutable, append-only transparency log that records all signatures
An OIDC provider: your identity source (Google, GitHub, etc.)

The Keyless Signing Flow (Step by Step):

For example when run cosign sign:

cosign sign kasisubbarayudu/buildkittest

Step 1: Ephemeral keypair generation

Cosign generates a keypair entirely in memory. The private key never touches disk. It exists only for this one signing operation.

Step 2: OIDC authentication

Cosign opens a browser and redirects you to the Sigstore OIDC broker (Dex), where you choose your identity provider, Google, GitHub, Microsoft, etc. You log in there.

 cosign sign kasisubbarayudu/buildkittest
WARNING: Image reference kasisubbarayudu/buildkittest uses a tag, not a digest, to identify the image to sign.
    This can lead you to sign a different image than the intended one. Please use a
    digest (example.com/ubuntu@sha256:abc123...) rather than tag
    (example.com/ubuntu:latest) for the input to cosign. The ability to refer to
    images by tag will be removed in a future release.

Generating ephemeral keys...

    The sigstore service, hosted by sigstore a Series of LF Projects, LLC, is provided pursuant to the Hosted Project Tools Terms of Use, available at https://lfprojects.org/policies/hosted-project-tools-terms-of-use/.
    Note that if your submission includes personal data associated with this signed artifact, it will be part of an immutable record.
    This may include the email address associated with the account with which you authenticate your contractual Agreement.
    This information will be used for signing this artifact and will be stored in public transparency logs and cannot be removed later, and is subject to the Immutable Record notice at https://lfprojects.org/policies/hosted-project-tools-immutable-records/.

By typing 'y', you attest that (1) you are not submitting the personal data of any other person; and (2) you understand and agree to the statement and the Agreement terms at the URLs listed above.
Are you sure you would like to continue? [y/N] y
Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore&code_challenge=plxge1Pb05zqWvd8VWJrQjKUlWK1ZrKm0hCk9BT1iZE&code_challenge_method=S256&nonce=TtZVuGy75XjszlKtZgAw7Q&redirect_uri=http%3A%2F%2Flocalhost%3A39883%2Fauth%2Fcallback&response_type=code&scope=openid+email&state=C4INaDJe_sTR7DOpJLWr6g
Opening in existing browser session.

Ephemeral keys are generated and kept in memory. As shown in the URL above, the link opens in the browser and connects to the auth endpoint of the DEX intermediate OIDC server. The response type is code, which means it uses the authorization code grant type with PKCE. Since Cosign is not a server-side application and runs on the user's device, it is not secure to store or embed the client ID or client secret in the Cosign binary. That is why the authorization code grant type with PKCE is used. In OIDC terms, cosign is considered a public client. OpenID scopes are specified to get more information about the user. Because these scopes are included, an ID token is returned along with the access token.

Here's something interesting: if you watch the browser URL carefully when you click "Sign in with Google," you'll notice the request goes to accounts.google.com, but the client_id belongs to Sigstore, and the redirect_uri points back to oauth2.sigstore.dev, not to your machine or cosign.

https://accounts.google.com/v3/signin/accountchooser?client_id=237800849078-hri2ndt7gdafpf34kq8crd5sik9pe3so.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Foauth2.sigstore.dev%2Fauth%2Fcallback&response_type=code&scope=openid+email&state=g57y3w6qy5ryk3u3q6bqekksp&dsh=S17075643%3A1774614974239725&o2v=2&service=lso&flowName=GeneralOAuthFlow&opparams=%253F&continue=https%3A%2F%2Faccounts.google.com%2Fsignin%2Foauth%2Fconsent%3Fauthuser%3Dunknown%26part%3DAJi8hANxu7AUaB6qU0RLCkwfiTl4NHKUzRTWxKZCL42fnEeigK80dX0JK9s3Uyi77N_lbNkMGEOjXsPmcD4C-3KKFFO1tgfnxveQAeqkz16cf8SWqwIrmMpMNBQieuwTqCWPgZf8SvkXEQ1c1mLhdHoarm438kgSAQINgb-GfWuvpu0DJKs9UoYyylZ2h0_h1Mk5mCZWmbEhm7Y8tAQjXe4pWfJiPZdsgld2Fuc-jADHCJnaypVIhVP4gCdSzlaOfdpl6oalYkJW5jtApJSExnjDIuuouSgksjbdw0_npecXC7hvHX5FDDjB6TCvcFo2q6SE8k8Lr2Ucj7zRWKmql3k7wdQvCZuUzaTLcM4-DucAvqDFuvbwhXwnCQzD2uSAwHb_VmRX5doPPW-aDY2juK0Y6P4lXfbwwttGaFupmCSvQIN8VjJuYdC43fTttfDD8M13CVZIddKqRicSrU5wPSpMYpRYXUysqg%26flowName%3DGeneralOAuthFlow%26as%3DS17075643%253A1774614974239725%26client_id%3D237800849078-hri2ndt7gdafpf34kq8crd5sik9pe3so.apps.googleusercontent.com%26requestPath%3D%252Fsignin%252Foauth%252Fconsent%23&app_domain=https%3A%2F%2Foauth2.sigstore.dev

client_id=237800849078-hri2ndt7gdafpf34kq8crd5sik9pe3so.apps.googleusercontent.com, This is Sigstore's own Google OAuth app ID. Sigstore registered itself as an app with Google. So Google sees Sigstore as the client, not you, not cosign.

After successful login:

Reason:

Many popular identity providers don't allow arbitrary values in the aud claim. For instance, Google OIDC populates the value from a registered OAuth 2.0 application's client ID and doesn't allow user-customizable values at all. Facebook and Microsoft do the same.

This is the core reason Dex exists. Fulcio needs to receive a token with aud=sigstore so it knows the token was intended for it. But Google will never let you set that audience claim, it's locked to Sigstore's own registered client ID. So Dex acts as the intermediary: your Google token goes to Dex (as Sigstore's registered Google OAuth client), Dex re-issues a new token with aud=sigstore, and that's what gets sent to Fulcio.

Once you authenticate, an OIDC id_token is produced and flows through this chain:

Google → Sigstore OAuth proxy → Cosign

Cosign also runs a small local HTTP server on localhost to receive the OAuth callback (the same pattern you'd see if you've ever implemented OIDC yourself). It uses the authorization code + PKCE flow.

Step 3: Certificate issuance from Fulcio

Cosign sends three things to Fulcio:

The OIDC id_token
The ephemeral public key
A signed challenge, a signature of the sub claim of the OIDC token, proving the ownership of the private key

Fulcio validates the OIDC token (checking it was actually signed by the OIDC provider), then proves that Cosign holds the private key by verifying the signature of the sub claim. Once both checks pass, Fulcio issues a short-lived X.509 certificate using its own CA, and this certificate is typically valid for 10 minutes, which binds your identity (the email from the OIDC token) to the ephemeral public key.

Step 4: Signing and uploading

Cosign takes the image digest (the sha256 hash), signs it with the ephemeral private key, and then uploads the signature, the certificate, and the transparency log entry to Rekor. The signature is also uploaded to the image registry (.sig artifact). After this, the private key is discarded.

Step 5: Verification

When anyone runs cosign verify:

Cosign fetches the .sig artifact from the registry. Cosign fetches the list of artifacts, such as SBOM, signature, and any other attestations attached to the image. Behind the scenes, it uses the referrers api of Docker. Among the returned artifacts, it tries to find the one that matches the artifactType: application/vnd.dev.sigstore.bundle.v0.3+json.

Ex: for image dockerhub.io/kasisubbarayudu/buildkittest,

curl -s "https://registry-1.docker.io/v2/kasisubbarayudu/buildkittest/referrers/sha256:20436a2b1237a073a1d8035342065b767923e7b884a41b049da8ca57f0ce7df2" \
  -H "Accept: application/vnd.oci.image.manifest.v1+json" \
  -H "Authorization: Bearer $(curl -s 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:kasisubbarayudu/buildkittest:pull' | jq -r .token)" \
  | jq .                                                   
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 889,
      "digest": "sha256:07dc49a44fb2ef30e67e2f50cf5485ae12b1b62ddb91cfc1fb24d9f604bd7689",
      "annotations": {
        "dev.sigstore.bundle.content": "dsse-envelope",
        "dev.sigstore.bundle.predicateType": "https://sigstore.dev/cosign/sign/v1",
        "org.opencontainers.image.created": "2026-06-08T07:05:26Z"
      },
      "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json"
    }
  ]
}

In the manifests we see one entry as other attestation for ex SBOM is not attached and only signature is attached.

You can use docker's manifest API, to get detailed info about the manifest:

curl -s "https://registry-1.docker.io/v2/kasisubbarayudu/buildkittest/manifests/sha256:07dc49a44fb2ef30e67e2f50cf5485ae12b1b62ddb91cfc1fb24d9f604bd7689" \
  -H "Accept: application/vnd.oci.image.manifest.v1+json" \
  -H "Authorization: Bearer $(curl -s 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:kasisubbarayudu/buildkittest:pull' | jq -r .token)" \
  | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.empty.v1+json",
    "size": 2,
    "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
    "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json"
  },
  "layers": [
    {
      "mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
      "size": 6867,
      "digest": "sha256:4e5a76ee7871b1349693013c23483305239cacd9e2e9b838aa6988a39ce65e3c"
    }
  ],
  "annotations": {
    "dev.sigstore.bundle.content": "dsse-envelope",
    "dev.sigstore.bundle.predicateType": "https://sigstore.dev/cosign/sign/v1",
    "org.opencontainers.image.created": "2026-06-08T07:05:26Z"
  },
  "subject": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "size": 2406,
    "digest": "sha256:20436a2b1237a073a1d8035342065b767923e7b884a41b049da8ca57f0ce7df2"
  },
  "artifactType": "application/vnd.dev.sigstore.bundle.v0.3+json"
}

And in the output, layers list contains one layer, whose sha value is "sha256:4e5a76ee7871b1349693013c23483305239cacd9e2e9b838aa6988a39ce65e3c" and is of type application/vnd.dev.sigstore.bundle.v0.3+json.

And if u try to get this blob using blob api:

curl \
  -H "Authorization: Bearer $(curl -s 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:kasisubbarayudu/buildkittest:pull' | jq -r .token)" \
  "https://registry-1.docker.io/v2/kasisubbarayudu/buildkittest/blobs/sha256:4e5a76ee7871b1349693013c23483305239cacd9e2e9b838aa6988a39ce65e3c" -L | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  6867  100  6867    0     0   6119      0  0:00:01  0:00:01 --:--:--  6119
{
  "mediaType": "application/vnd.dev.sigstore.bundle.v0.3+json",
  "verificationMaterial": {
    "certificate": {
      "rawBytes": "MIIDMDCCArWgAwIBAgIUDQfrQyDE+a0p1AGOeILRzKPdoFswCgYIKoZIzj0EAwMwNzEVMBMGA1UEChMMc2lnc3RvcmUuZGV2MR4wHAYDVQQDExVzaWdzdG9yZS1pbnRlcm1lZGlhdGUwHhcNMjYwNjA4MDcwNTI1WhcNMjYwNjA4MDcxNTI1WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEwmo8XT1QN0I66R6shMuxGVfiS4/DKjrUB38+5OxC1fE1r/VCa2dt/hvDXO+MhmaKjooAw40Fuk5ePyvlzbeuWqOCAdQwggHQMA4GA1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAzAdBgNVHQ4EFgQUN3GvpnBI8TR3+HPngQojmxxAkP4wHwYDVR0jBBgwFoAU39Ppz1YkEZb5qNjpKFWixi4YZD8wJwYDVR0RAQH/BB0wG4EZa2FzaXJhbWtvbXBlbGxhQGdtYWlsLmNvbTApBgorBgEEAYO/MAEBBBtodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20wKwYKKwYBBAGDvzABCAQdDBtodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20wWwYKKwYBBAGDvzABGARNDEtDaFV4TURBMk16WTBPVGc1TWpReE1qQXpOVE0wT0RjU0gyaDBkSEJ6T2lVeVJpVXlSbUZqWTI5MWJuUnpMbWR2YjJkc1pTNWpiMjAwgYoGCisGAQQB1nkCBAIEfAR6AHgAdgDdPTBqxscRMmMZHhyZZzcCokpeuN48rf+HinKALynujgAAAZ6mDOyqAAAEAwBHMEUCIQCpgLPskeYhfJtvkTZYKEDm+Wvr1wbE6r0GYxA7YAe1igIgNWMh/QYATTjbIUB35BJivXA3ptzlwZAOR9si3uhkRqswCgYIKoZIzj0EAwMDaQAwZgIxALMQKyyQXlXc1cwBbhgakQwoBgbIF0eftEEmIkzclOotX8j6v79D0jkbPfmVDAr+MAIxALya8kp3WQarAgotAasne32EQcBm8MFEI4ClIyMoyaKUkzFYqkUYfBwX6IkdXNS/HA=="
    },
    "tlogEntries": [
      {
        "logIndex": "1754783759",
        "logId": {
          "keyId": "wNI9atQGlz+VWfO6LRygH4QUfY/8W4RFwiT5i5WRgB0="
        },
        "kindVersion": {
          "kind": "dsse",
          "version": "0.0.1"
        },
        "integratedTime": "1780902326",
        "inclusionPromise": {
          "signedEntryTimestamp": "MEQCIEeby7uJfKo1ZqR/Z8clj8942TRqn9FeZtqWnoLZOBMtAiBYJBPc1Hkffz12y5d9l0sLQbu0ZZQ2NOB38I6gMAsWFQ=="
        },
        "inclusionProof": {
          "logIndex": "1632879497",
          "rootHash": "hQVgleW4096bWB5nMVA4euGOFNXspkZM7fkwOU0DRTo=",
          "treeSize": "1632879500",
          "hashes": [
            "XHIMw0aYbwSRZIDloHHhezy6eFMxDuBywnaCAEZ/hCE=",
            "47B8Brg1p3ZJ5Q20oIZDiAdFEs/Eg1bJ4T9EkyqGnwU=",
            "PCTg9y1uHHEyzk0yrF+0KIvYlf739uZWx8mLPkD4qWM=",
            "IG3DXh2vVc0tB/VXEUb49zN6VKXFgDbkYFjSbEoLnis=",
            "/6gQW6oyjPBJt9hHaloGnKe/cqwTgDP+ehCA17xJDnE=",
            "4OW3KC98wj7ppZzIz78pbAZfUPaNQOFcX/+X1RWAqEk=",
            "rrZJWna+MzeY9FZ/DOBhYRvAmyLC+usz7+CSc7D6ABA=",
            "EqM3uHc9tMy5EM7eybA0+bnEzNFb6bYf02wL+9dv7qc=",
            "AKwEP2HH5SGxOZY6n7sb0sQ8/KMJRFtSn3nT/co5fZY=",
            "RjW1XPhqoinfsobGd20S3gJis0k6WUSEpjaQ6JXQ5FI=",
            "BhdMVwa7xxwCWC//5qOnv/wqs69cjw8nHrPed7oJkAI=",
            "9jo43+euK7XyKWxBn2h5BwzuDA8PaUq2geJ8WGWycpQ=",
            "x5jI42lOBdSMG7JZOSaglWmLNIaPCZTz6pXBlb7ZgYQ=",
            "daxmZaajRpZV+JxHiOYZhJBiSKN5ucqjh2WnGbHhirw=",
            "DOCeoSMovIvLExkhIvisow9AuNXgeWs4ECkyR6EcqYU="
          ],
          "checkpoint": {
            "envelope": "rekor.sigstore.dev - 1193050959916656506\n1632879500\nhQVgleW4096bWB5nMVA4euGOFNXspkZM7fkwOU0DRTo=\n\n— rekor.sigstore.dev wNI9ajBGAiEApRMPDWfsFq4rhLkDJBbREL2Wc5ZAqxWaGMylgm23WfUCIQDsyP5tohOidsES6Hb9KWdS0xzaoPzuMIPslnSS010XhA==\n"
          }
        },
        "canonicalizedBody": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiZHNzZSIsInNwZWMiOnsiZW52ZWxvcGVIYXNoIjp7ImFsZ29yaXRobSI6InNoYTI1NiIsInZhbHVlIjoiOTE5ZjYyNzM3OWZiYWM3NjUxODc5ZWRiZTdhN2NhMDM2OTVjODA4YjFkODk2MThmNjhhNjNlM2VmNDY5OTAyZCJ9LCJwYXlsb2FkSGFzaCI6eyJhbGdvcml0aG0iOiJzaGEyNTYiLCJ2YWx1ZSI6ImU1NDI4OTc5NGFkZGQyYWU5NDVkODc2NDY0MzRiMTMzZjE1MjU0ODFiMDUxMTAzZmIyN2YxNDQxYWU3NjY0YzkifSwic2lnbmF0dXJlcyI6W3sic2lnbmF0dXJlIjoiTUVVQ0lDTmFzY3N1MTNtcGRjTG5ET21KMkNEc25TYW9QbHFSR3RGOFlyMjBsVkZwQWlFQXZ1L3RCVFArWTJuOFlyYXRnUFoveE5ZSkc2RWpNaGoxckQ3VU9pdlNVRjg9IiwidmVyaWZpZXIiOiJMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSTlJFTkRRWEpYWjBGM1NVSkJaMGxWUkZGbWNsRjVSRVVyWVRCd01VRkhUMlZKVEZKNlMxQmtiMFp6ZDBObldVbExiMXBKZW1vd1JVRjNUWGNLVG5wRlZrMUNUVWRCTVZWRlEyaE5UV015Ykc1ak0xSjJZMjFWZFZwSFZqSk5ValIzU0VGWlJGWlJVVVJGZUZaNllWZGtlbVJIT1hsYVV6RndZbTVTYkFwamJURnNXa2RzYUdSSFZYZElhR05PVFdwWmQwNXFRVFJOUkdOM1RsUkpNVmRvWTA1TmFsbDNUbXBCTkUxRVkzaE9WRWt4VjJwQlFVMUdhM2RGZDFsSUNrdHZXa2w2YWpCRFFWRlpTVXR2V2tsNmFqQkVRVkZqUkZGblFVVjNiVzg0V0ZReFVVNHdTVFkyVWpaemFFMTFlRWRXWm1sVE5DOUVTMnB5VlVJek9Dc0tOVTk0UXpGbVJURnlMMVpEWVRKa2RDOW9ka1JZVHl0TmFHMWhTMnB2YjBGM05EQkdkV3MxWlZCNWRteDZZbVYxVjNGUFEwRmtVWGRuWjBoUlRVRTBSd3BCTVZWa1JIZEZRaTkzVVVWQmQwbElaMFJCVkVKblRsWklVMVZGUkVSQlMwSm5aM0pDWjBWR1FsRmpSRUY2UVdSQ1owNVdTRkUwUlVablVWVk9NMGQyQ25CdVFrazRWRkl6SzBoUWJtZFJiMnB0ZUhoQmExQTBkMGgzV1VSV1VqQnFRa0puZDBadlFWVXpPVkJ3ZWpGWmEwVmFZalZ4VG1wd1MwWlhhWGhwTkZrS1drUTRkMHAzV1VSV1VqQlNRVkZJTDBKQ01IZEhORVZhWVRKR2VtRllTbWhpVjNSMllsaENiR0pIZUdoUlIyUjBXVmRzYzB4dFRuWmlWRUZ3UW1kdmNncENaMFZGUVZsUEwwMUJSVUpDUW5SdlpFaFNkMk42YjNaTU1rWnFXVEk1TVdKdVVucE1iV1IyWWpKa2MxcFROV3BpTWpCM1MzZFpTMHQzV1VKQ1FVZEVDblo2UVVKRFFWRmtSRUowYjJSSVVuZGplbTkyVERKR2Fsa3lPVEZpYmxKNlRHMWtkbUl5WkhOYVV6VnFZakl3ZDFkM1dVdExkMWxDUWtGSFJIWjZRVUlLUjBGU1RrUkZkRVJoUmxZMFZGVlNRazFyTVRaWFZFSlFWa2RqTVZSWGNGSmxSVEZ4VVZod1QxWkZNSGRVTUZKcVZUQm5lV0ZFUW10VFJVbzJWREpzVmdwbFZrcHdWbGhzVTJKVlduRlhWRWsxVFZkS2RWVnVjRTFpVjFJeVdXcEthMk14Y0ZST1YzQnBUV3BCZDJkWmIwZERhWE5IUVZGUlFqRnVhME5DUVVsRkNtWkJValpCU0dkQlpHZEVaRkJVUW5GNGMyTlNUVzFOV2tob2VWcGFlbU5EYjJ0d1pYVk9ORGh5Wml0SWFXNUxRVXg1Ym5WcVowRkJRVm8yYlVSUGVYRUtRVUZCUlVGM1FraE5SVlZEU1ZGRGNHZE1VSE5yWlZsb1prcDBkbXRVV2xsTFJVUnRLMWQyY2pGM1lrVTJjakJIV1hoQk4xbEJaVEZwWjBsblRsZE5hQW92VVZsQlZGUnFZa2xWUWpNMVFrcHBkbGhCTTNCMGVteDNXa0ZQVWpsemFUTjFhR3RTY1hOM1EyZFpTVXR2V2tsNmFqQkZRWGROUkdGUlFYZGFaMGw0Q2tGTVRWRkxlWGxSV0d4WVl6RmpkMEppYUdkaGExRjNiMEpuWWtsR01HVm1kRVZGYlVscmVtTnNUMjkwV0RocU5uWTNPVVF3YW10aVVHWnRWa1JCY2lzS1RVRkplRUZNZVdFNGEzQXpWMUZoY2tGbmIzUkJZWE51WlRNeVJWRmpRbTA0VFVaRlNUUkRiRWw1VFc5NVlVdFZhM3BHV1hGclZWbG1RbmRZTmtsclpBcFlUbE12U0VFOVBRb3RMUzB0TFVWT1JDQkRSVkpVU1VaSlEwRlVSUzB0TFMwdENnPT0ifV19fQ=="
      }
    ],
    "timestampVerificationData": {
      "rfc3161Timestamps": [
        {
          "signedTimestamp": "MIICyTADAgEAMIICwAYJKoZIhvcNAQcCoIICsTCCAq0CAQMxDTALBglghkgBZQMEAgEwgbgGCyqGSIb3DQEJEAEEoIGoBIGlMIGiAgEBBgkrBgEEAYO/MAIwMTANBglghkgBZQMEAgEFAAQgu2VISVNbYk+VQZDi6yCnss5LGGVpFuknPHtFsktL4OECFQD6OOcltBBcOCkve+XSgeOBgmWURBgPMjAyNjA2MDgwNzA1MjVaMAMCAQGgMqQwMC4xFTATBgNVBAoTDHNpZ3N0b3JlLmRldjEVMBMGA1UEAxMMc2lnc3RvcmUtdHNhoAAxggHaMIIB1gIBATBRMDkxFTATBgNVBAoTDHNpZ3N0b3JlLmRldjEgMB4GA1UEAxMXc2lnc3RvcmUtdHNhLXNlbGZzaWduZWQCFDoTVC8MkGHuvMFDL8uKjosqI4sMMAsGCWCGSAFlAwQCAaCB/DAaBgkqhkiG9w0BCQMxDQYLKoZIhvcNAQkQAQQwHAYJKoZIhvcNAQkFMQ8XDTI2MDYwODA3MDUyNVowLwYJKoZIhvcNAQkEMSIEIAZYYA1ib9CRgqXzK1RqCxC+FINsWTmm2Bd5EOH9v7n6MIGOBgsqhkiG9w0BCRACLzF/MH0wezB5BCCF+Se8B6tiysO0Q1bBDvyBssaIP9p6uebYcNnROs0FtzBVMD2kOzA5MRUwEwYDVQQKEwxzaWdzdG9yZS5kZXYxIDAeBgNVBAMTF3NpZ3N0b3JlLXRzYS1zZWxmc2lnbmVkAhQ6E1QvDJBh7rzBQy/Lio6LKiOLDDAKBggqhkjOPQQDAgRmMGQCMBspPBil7OP6OoSS1pPrreov6kqG/blObbEfqvdkcqwlet9pKNcGkrm/1AdgPu0fsAIwfUfRv0eF497BMcGmeO1SEmBxXMWj9VJQt0kp+IYAB+8wSD0n3o8wmY+sQ2FFczSL"
        }
      ]
    }
  },
  "dsseEnvelope": {
    "payload": "eyJfdHlwZSI6Imh0dHBzOi8vaW4tdG90by5pby9TdGF0ZW1lbnQvdjEiLCAic3ViamVjdCI6W3siZGlnZXN0Ijp7InNoYTI1NiI6IjIwNDM2YTJiMTIzN2EwNzNhMWQ4MDM1MzQyMDY1Yjc2NzkyM2U3Yjg4NGE0MWIwNDlkYThjYTU3ZjBjZTdkZjIifSwgImFubm90YXRpb25zIjp7fX1dLCAicHJlZGljYXRlVHlwZSI6Imh0dHBzOi8vc2lnc3RvcmUuZGV2L2Nvc2lnbi9zaWduL3YxIiwgInByZWRpY2F0ZSI6e319",
    "payloadType": "application/vnd.in-toto+json",
    "signatures": [
      {
        "sig": "MEUCICNascsu13mpdcLnDOmJ2CDsnSaoPlqRGtF8Yr20lVFpAiEAvu/tBTP+Y2n8YratgPZ/xNYJG6EjMhj1rD7UOivSUF8="
      }
    ]
  }
}

It returns the JSON response since the layer mediaType was application/vnd.dev.sigstore.bundle.v0.3+json json.

When cosign verify is run, for ex:

cosign verify kasisubbarayudu/buildkittest   --certificate-oidc-issuer https://accounts.google.com   --certificate-identity  kasisubbarayudu@gmail.com

cosign does something similar: it fetches the JSON bundle from the signature artifact, then decodes the RAW certificate bytes, then extracts the Subject Alternative Name, checks whether it matches the --certificate-identity.

Decoded cert:

echo "MIIDMDCCArWgAwIBAgIUDQfrQyDE+a0p1AGOeILRzKPdoFswCgYIKoZIzj0EAwMwNzEVMBMGA1UEChMMc2lnc3RvcmUuZGV2MR4wHAYDVQQDExVzaWdzdG9yZS1pbnRlcm1lZGlhdGUwHhcNMjYwNjA4MDcwNTI1WhcNMjYwNjA4MDcxNTI1WjAAMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEwmo8XT1QN0I66R6shMuxGVfiS4/DKjrUB38+5OxC1fE1r/VCa2dt/hvDXO+MhmaKjooAw40Fuk5ePyvlzbeuWqOCAdQwggHQMA4GA1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAzAdBgNVHQ4EFgQUN3GvpnBI8TR3+HPngQojmxxAkP4wHwYDVR0jBBgwFoAU39Ppz1YkEZb5qNjpKFWixi4YZD8wJwYDVR0RAQH/BB0wG4EZa2FzaXJhbWtvbXBlbGxhQGdtYWlsLmNvbTApBgorBgEEAYO/MAEBBBtodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20wKwYKKwYBBAGDvzABCAQdDBtodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20wWwYKKwYBBAGDvzABGARNDEtDaFV4TURBMk16WTBPVGc1TWpReE1qQXpOVE0wT0RjU0gyaDBkSEJ6T2lVeVJpVXlSbUZqWTI5MWJuUnpMbWR2YjJkc1pTNWpiMjAwgYoGCisGAQQB1nkCBAIEfAR6AHgAdgDdPTBqxscRMmMZHhyZZzcCokpeuN48rf+HinKALynujgAAAZ6mDOyqAAAEAwBHMEUCIQCpgLPskeYhfJtvkTZYKEDm+Wvr1wbE6r0GYxA7YAe1igIgNWMh/QYATTjbIUB35BJivXA3ptzlwZAOR9si3uhkRqswCgYIKoZIzj0EAwMDaQAwZgIxALMQKyyQXlXc1cwBbhgakQwoBgbIF0eftEEmIkzclOotX8j6v79D0jkbPfmVDAr+MAIxALya8kp3WQarAgotAasne32EQcBm8MFEI4ClIyMoyaKUkzFYqkUYfBwX6IkdXNS/HA==" | base64 -d | openssl x509 -inform DER -text -noout 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0d:07:eb:43:20:c4:f9:ad:29:d4:01:8e:78:82:d1:cc:a3:dd:a0:5b
        Signature Algorithm: ecdsa-with-SHA384
        Issuer: O = sigstore.dev, CN = sigstore-intermediate
        Validity
            Not Before: Jun  8 07:05:25 2026 GMT
            Not After : Jun  8 07:15:25 2026 GMT
        Subject: 
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:c2:6a:3c:5d:3d:50:37:42:3a:e9:1e:ac:84:cb:
                    b1:19:57:e2:4b:8f:c3:2a:3a:d4:07:7f:3e:e4:ec:
                    42:d5:f1:35:af:f5:42:6b:67:6d:fe:1b:c3:5c:ef:
                    8c:86:66:8a:8e:8a:00:c3:8d:05:ba:4e:5e:3f:2b:
                    e5:cd:b7:ae:5a
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage: 
                Code Signing
            X509v3 Subject Key Identifier: 
                37:71:AF:A6:70:48:F1:34:77:F8:73:E7:81:0A:23:9B:1C:40:90:FE
            X509v3 Authority Key Identifier: 
                DF:D3:E9:CF:56:24:11:96:F9:A8:D8:E9:28:55:A2:C6:2E:18:64:3F
            X509v3 Subject Alternative Name: critical
                email:kasisubbarayudu@gmail.com
            1.3.6.1.4.1.57264.1.1: 
                https://accounts.google.com
            1.3.6.1.4.1.57264.1.8: 
                ..https://accounts.google.com
            1.3.6.1.4.1.57264.1.24: 
                .KChUxMDA2MzY0OTg5MjQxMjAzNTM0ODcSH2h0dHBzOiUyRiUyRmFjY291bnRzLmdvb2dsZS5jb20
            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : DD:3D:30:6A:C6:C7:11:32:63:19:1E:1C:99:67:37:02:
                                A2:4A:5E:B8:DE:3C:AD:FF:87:8A:72:80:2F:29:EE:8E
                    Timestamp : Jun  8 07:05:25.418 2026 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:A9:80:B3:EC:91:E6:21:7C:9B:6F:91:
                                36:58:28:40:E6:F9:6B:EB:D7:06:C4:EA:BD:06:63:10:
                                3B:60:07:B5:8A:02:20:35:63:21:FD:06:00:4D:38:DB:
                                21:40:77:E4:12:62:BD:70:37:A6:DC:E5:C1:90:0E:47:
                                DB:22:DE:E8:64:46:AB
    Signature Algorithm: ecdsa-with-SHA384
    Signature Value:
        30:66:02:31:00:b3:10:2b:2c:90:5e:55:dc:d5:cc:01:6e:18:
        1a:91:0c:28:06:06:c8:17:47:9f:b4:41:26:22:4c:dc:94:ea:
        2d:5f:c8:fa:bf:bf:43:d2:39:1b:3d:f9:95:0c:0a:fe:30:02:
        31:00:bc:9a:f2:4a:77:59:06:ab:02:0a:2d:01:ab:27:7b:7d:
        84:41:c0:66:f0:c1:44:23:80:a5:23:23:28:c9:a2:94:93:31:
        58:aa:45:18:7c:1c:17:e8:89:1d:5c:d4:bf:1c

Also checks if issuer matches --certificate-oidc-issuer.

It also verifies if the certificate is really signed by the Fulcio CA.

Most importantly, it validates the signature of the image. It decodes the dsseEnvelope.payload from the JSON response. This payload contains the image digest that was signed.

In our case, for ex:

echo "eyJfdHlwZSI6Imh0dHBzOi8vaW4tdG90by5pby9TdGF0ZW1lbnQvdjEiLCAic3ViamVjdCI6W3siZGlnZXN0Ijp7InNoYTI1NiI6IjIwNDM2YTJiMTIzN2EwNzNhMWQ4MDM1MzQyMDY1Yjc2NzkyM2U3Yjg4NGE0MWIwNDlkYThjYTU3ZjBjZTdkZjIifSwgImFubm90YXRpb25zIjp7fX1dLCAicHJlZGljYXRlVHlwZSI6Imh0dHBzOi8vc2lnc3RvcmUuZGV2L2Nvc2lnbi9zaWduL3YxIiwgInByZWRpY2F0ZSI6e319" | base64 -d
{"_type":"https://in-toto.io/Statement/v1", "subject":[{"digest":{"sha256":"20436a2b1237a073a1d8035342065b767923e7b884a41b049da8ca57f0ce7df2"}, "annotations":{}}], "predicateType":"https://sigstore.dev/cosign/sign/v1", "predicate":{}}

And then it validates the signature dsseEnvelope.signatures.sig using public key from certificate.

Here, the certificate is valid for 10 minutes only. The Rekor log entry is the durable record, not the certificate's validity period.

Keyless Signing in GitHub Actions

The flow above works for human-initiated signing from a laptop. But in CI/CD pipelines, there's no browser, no human to click through an OAuth flow. This is where GitHub Actions' OIDC integration comes in.

How It Works?

GitHub Actions can generate OIDC tokens for workflow runs. These tokens identify the workflow itself, the repository, the ref, the workflow file path, rather than a human email address. To enable this, your workflow needs:

permissions:
  contents: read
  id-token: write

The id-token: write permission tells GitHub to inject two environment variables into the runner:

ACTIONS_ID_TOKEN_REQUEST_URL: the GitHub OIDC endpoint
ACTIONS_ID_TOKEN_REQUEST_TOKEN: a bearer token to authenticate with that endpoint (not the final OIDC token itself — this is a common point of confusion)

When cosign runs in the pipeline, it internally does something like this:

GET $ACTIONS_ID_TOKEN_REQUEST_URL?audience=sigstore
Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN

GitHub responds with an actual OIDC id_token whose aud claim is set to sigstore. From there, the flow is identical to the keyless human flow: cosign sends this token plus the ephemeral public key to Fulcio, gets a certificate, signs the image digest, and uploads to Rekor.
Again, since Google doesn't support setting custom aud claims, a Dex intermediate OIDC server was needed, but in the case of GitHub actions, since it allows token creation with a custom audience, a Dex server isn't needed.

Sample workflow:

name: Build, Sign, Push to DockerHub

on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  build-sign-push:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Source Code
        uses: actions/checkout@v4

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to DockerHub
        uses: docker/login-action@v3
        with:
          registry: docker.io
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and Push Docker Image
        id: build-push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ./Dockerfile
          push: true
          tags: docker.io/${{ secrets.DOCKERHUB_USERNAME }}/myapp:${{ github.sha }}

      - name: Sign Image (Keyless)
        run: |
          cosign sign --yes \
            docker.io/${{ secrets.DOCKERHUB_USERNAME }}/myapp@${{ steps.build-push.outputs.digest }}

      - name: Verify Signature
        run: |
          cosign verify \
            --certificate-identity "https://github.com/${{ github.repository }}/.github/workflows/workflow.yaml@refs/heads/main" \
            --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
            docker.io/${{ secrets.DOCKERHUB_USERNAME }}/myapp@${{ steps.build-push.outputs.digest }}

One thing to note here:

The certificate identity is the workflow path. When verifying an Actions-signed image, the --certificate-identity isn't an email; it's the fully-qualified workflow file URL. The Fulcio certificate for a GitHub Actions signing looks like:

data:
  Serial Number: '0x05f43452ef80bc88abeda2b2495909fa1236ce61'
Signature:
  Issuer: O=sigstore.dev, CN=sigstore-intermediate
  Validity:
    Not Before: 15 days ago (2026-04-02T20:06:17+05:30)
    Not After: 15 days ago (2026-04-02T20:16:17+05:30)
  Algorithm:
    name: ECDSA
    namedCurve: P-256
  Subject:
    extraNames:
      items: {}
    asn: []
X509v3 extensions:
  Key Usage (critical):
  - Digital Signature
  Extended Key Usage:
  - Code Signing
  Subject Key Identifier:
  - C0:B2:07:6D:68:73:B1:57:3D:DC:E1:A5:7B:4A:ED:E1:9B:8E:6C:21
  Authority Key Identifier:
    keyid: DF:D3:E9:CF:56:24:11:96:F9:A8:D8:E9:28:55:A2:C6:2E:18:64:3F
  Subject Alternative Name (critical):
    url:
    - https://github.com/srisaikompella/pr-test/.github/workflows/workflow.yaml@refs/heads/main
  OIDC Issuer: https://token.actions.githubusercontent.com
  GitHub Workflow Trigger: push
  GitHub Workflow SHA: fe04ec0ce732b6be9fc1446333eceb111bfb0daf
  GitHub Workflow Name: Build, Sign, Push to DockerHub
  GitHub Workflow Repository: srisaikompella/pr-test
  GitHub Workflow Ref: refs/heads/main
  OIDC Issuer (v2): https://token.actions.githubusercontent.com
  Build Signer URI: https://github.com/srisaikompella/pr-test/.github/workflows/workflow.yaml@refs/heads/main
  Build Signer Digest: fe04ec0ce732b6be9fc1446333eceb111bfb0daf
  Runner Environment: github-hosted
  Source Repository URI: https://github.com/srisaikompella/pr-test
  Source Repository Digest: fe04ec0ce732b6be9fc1446333eceb111bfb0daf
  Source Repository Ref: refs/heads/main
  Source Repository Identifier: '605870062'
  Source Repository Owner URI: https://github.com/srisaikompella
  Source Repository Owner Identifier: '69502069'
  Build Config URI: https://github.com/srisaikompella/pr-test/.github/workflows/workflow.yaml@refs/heads/main
  Build Config Digest: fe04ec0ce732b6be9fc1446333eceb111bfb0daf
  Build Trigger: push
  Run Invocation URI: https://github.com/srisaikompella/pr-test/actions/runs/23905840052/attempts/1
  Source Repository Visibility At Signing: public
  1.3.6.1.4.1.11129.2.4.2: 04:7a:00:78:00:76:00:dd:3d:30:6a:c6:c7:11:32:63:19:1e:1c:99:67:37:02:a2:4a:5e:b8:de:3c:ad:ff:87:8a:72:80:2f:29:ee:8e:00:00:01:9d:4e:9f:a0:9d:00:00:04:03:00:47:30:45:02:20:14:74:26:99:7a:9a:07:ba:a0:e9:a6:b2:cc:bf:7a:8b:53:6b:4b:d9:c8:d1:74:ec:61:13:df:b8:9b:46:e1:5d:02:21:00:96:4e:cd:77:2d:e4:00:6c:1c:cf:a1:bd:72:d3:4a:7b:32:8f:29:94:da:f6:1b:48:f9:6e:d0:5e:5e:78:be:85

Final Thoughts

Signing images is only half the story. The other half is making sure your cluster actually enforces it, rejecting any workload whose image wasn't signed. In the next article, we will setup sigstore policy controller that blocks unsigned images.

Provisioning a Kubernetes Cluster on OpenStack Using Cluster API (CAPI)

Sri Sai Venkata Kasi Subbarayudu Kompella — Sun, 29 Mar 2026 13:24:25 +0000

This is my first blog post. I spent this weekend going deep into CAPI + CAPO + ORC. This guide covers everything - installation, configuration, manifests, and the full flow explained simply. If you want to understand CAPI, this guide is for you.

What Is Cluster API and Why Should You Care?

Managing Kubernetes clusters manually is painful. You manually create infrastructure, install Kubernetes packages, SSH into VMs, run kubeadm init, and then run kubeadm join to join the worker nodes, set up CNI, etc. Cluster API (CAPI) solves this by treating cluster creation as a Kubernetes-native, declarative operation. Cluster API is a Kubernetes project that lets you manage Kubernetes clusters using Kubernetes-style APIs.

Instead of running commands, you write YAML manifests and apply them to a "management cluster." CAPI controllers read those manifests and provision your cluster automatically — creating VMs, configuring networking, running kubeadm — all without you touching a single VM.

Three core concepts to understand:

Management Cluster — A small existing Kubernetes cluster that runs CAPI controllers and manages other clusters
Workload Cluster — The cluster CAPI creates for you, where your actual apps run
Providers — Plugins that tell CAPI how to talk to specific infrastructure (OpenStack, AWS, GCP, etc.)

Role of controllers:

CAPI installs multiple controllers.

Controller	Role
CAPI Core	Orchestrates cluster lifecycle — watches Cluster, Machine, MachineDeployment objects
CAPO (Cluster API Provider OpenStack)	Talks to OpenStack APIs — creates Nova VMs, Octavia LBs
CABPK (Kubeadm Bootstrap Provider)	Generates cloud-init scripts that run kubeadm on VMs
KCP (Kubeadm Control Plane Provider)	Manages control plane machine lifecycle and scaling

All are installed together via clusterctl.

If you have been reading about CAPO, you may have come across ORC — the OpenStack Resource Controller. ORC is a separate project and is not part of CAPO. It needs to be installed separately. ORC is a separate, optional Kubernetes controller that lets you manage OpenStack resources — Glance images, Nova keypairs, Neutron networks and routers — as Kubernetes custom resources, declaratively. The idea is that instead of manually uploading your node image to Glance and copying its UUID into your CAPI manifests, you declare an Image CRD in your management cluster, and ORC ensures it exists in OpenStack. It ensures the resources referenced in the CAPI manifest exist before creating the cluster.

Step 1 — Prerequisites

Before installing CAPI, you need:

A running management cluster (even a single-node kind or minikube cluster works)
clusterctl CLI installed
OpenStack credentials (your clouds.yaml file)
OpenStack environment with:
- At least one available flavor (e.g., m1.medium)
- A Glance image with Kubernetes pre-installed (qcow image with kubeadm/kubelet/kubectl pre-installed)
- A provider network created
- Octavia (Load Balancer service) enabled

Install clusterctl:

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 \
  -o clusterctl
chmod +x clusterctl
sudo mv clusterctl /usr/local/bin/

Verify:

clusterctl version

Sample output:

clusterctl version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"1a1852c74072febc3fbf9823c9dd32f4cd6cc747", GitTreeState:"clean", BuildDate:"2026-02-17T17:46:30Z", GoVersion:"go1.24.13", Compiler:"gc", Platform:"linux/amd64"}

Step 2 — Initialize CAPI with OpenStack Provider

clusterctl init --infrastructure openstack:v0.14.1

This installs:

CAPI core controllers
CAPO (OpenStack infrastructure provider)
CABPK (kubeadm bootstrap provider)
KCP (kubeadm control plane provider)

Sample Output:

Fetching providers
Installing cert-manager version="v1.19.3"
Waiting for cert-manager to be available...
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="cluster-api" version="v1.12.4" targetNamespace="capi-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="bootstrap-kubeadm" version="v1.12.4" targetNamespace="capi-kubeadm-bootstrap-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="control-plane-kubeadm" version="v1.12.4" targetNamespace="capi-kubeadm-control-plane-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Installing provider="infrastructure-openstack" version="v0.14.1" targetNamespace="capo-system"
spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`.
Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Verify all controllers are running:

kubectl get pods -A | grep -E "capi|capo|cert-manager"
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-5b6b785789-r62sl      1/1     Running   0          42m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-95df764cf-hh7km   1/1     Running   0          42m
capi-system                         capi-controller-manager-6bbbfd79d8-b2dm4                        1/1     Running   0          42m
capo-system                         capo-controller-manager-6469cf765b-xhv8s                        1/1     Running   0          42m
cert-manager                        cert-manager-77f7698dcf-76hsq                                   1/1     Running   0          43m
cert-manager                        cert-manager-cainjector-557b97dd67-52fnw                        1/1     Running   0          43m
cert-manager                        cert-manager-webhook-5b7654ff4-cmgc8                            1/1     Running   0          43m

Step 3 — Set Environment Variables for Template Generation

clusterctl can generate cluster manifests from a template. You need to export these environment variables first:

export OPENSTACK_CLOUD=openstack # name should match the cloudname in clouds.yaml
export OPENSTACK_CLOUD_CACERT_B64=""
export  OPENSTACK_CLOUD_YAML_B64=$(base64 -w0 clouds.yaml) # get clouds.yaml from openstack horizon dashboard
export OPENSTACK_NODE_MACHINE_FLAVOR=m1.medium # instance type for ctrl plane node
export OPENSTACK_IMAGE_NAME=ubuntu-2404-kube # Name of the image used for kubernetes cluster.
export OPENSTACK_FAILURE_DOMAIN=nova  # avail zone usually nova.
export OPENSTACK_EXTERNAL_NETWORK_ID=5068d3a6-f560-4d79-b834-cd6943dc5de5 # ex: openstack network list | grep provider, get the id.
export OPENSTACK_DNS_NAMESERVERS=8.8.8.8
export OPENSTACK_SSH_KEY_NAME="kashi" # name of the SSH key uploaded in OpenStack and you want to be uploaded to cluster nodes

The image used in CAPI is prebuilt image that contains kubeadm, kubectl, kubelet all the kubernetes packages preinstalled.

You can build the image by running the following Makefile:

SHELL := /bin/bash

IMAGE_BUILDER_REPO=https://github.com/kubernetes-sigs/image-builder.git
IMAGE_BUILDER_DIR=image-builder

ifndef IMAGE_NAME
IMAGE_NAME=ubuntu-2404-kube-v1.34.3
$(warning IMAGE_NAME is undefined, using default: $(IMAGE_NAME))
endif

ifndef IMAGE_FILE
IMAGE_FILE=output/ubuntu-2404-kube-v1.34.3/ubuntu-2404-kube-v1.34.3
$(warning IMAGE_FILE is undefined, using default: $(IMAGE_FILE))
endif

OS_IMAGE_NAME=$(IMAGE_NAME)
OS_DISK_FORMAT=qcow2
OS_CONTAINER_FORMAT=bare
OS_VISIBILITY=public
ifndef ADMIN_RC_PATH
ADMIN_RC_PATH=/etc/kolla/admin-openrc.sh
$(warning ADMIN_RC_PATH is undefined, using default: $(ADMIN_RC_PATH))
endif


img-builder-clone:
    if [ ! -d "$(IMAGE_BUILDER_DIR)" ]; then \
        git clone $(IMAGE_BUILDER_REPO); \
    fi

capi-deps: img-builder-clone
    cd $(IMAGE_BUILDER_DIR)/images/capi && make deps-qemu


build-capi: capi-deps
    if [ ! -f "$(IMAGE_BUILDER_DIR)/images/capi/$(IMAGE_FILE)" ]; then \
       cd $(IMAGE_BUILDER_DIR)/images/capi && make build-qemu-ubuntu-2404 &> /dev/null; \
    fi


capi-upload: 
     . $(ADMIN_RC_PATH) && openstack image create $(OS_IMAGE_NAME) \
        --file $(IMAGE_BUILDER_DIR)/images/capi/$(IMAGE_FILE) \
        --disk-format $(OS_DISK_FORMAT) \
        --container-format $(OS_CONTAINER_FORMAT) \
        --public


capi-image: build-capi capi-upload

capi-delete:
     . $(ADMIN_RC_PATH) && openstack image delete $(OS_IMAGE_NAME)

all: capi-image

clean: 
    rm -rf $(IMAGE_BUILDER_DIR)  $(DISKIMAGE_DIR) $(OCTAVIA_REPO)

Run the makefile: make all

You can list the required environment variables to be set by running the following command:

clusterctl generate cluster --infrastructure openstack:v0.14.1 --list-variables capi-quickstart

Sample Output:

Required Variables:
  - KUBERNETES_VERSION
  - OPENSTACK_CLOUD
  - OPENSTACK_CLOUD_CACERT_B64
  - OPENSTACK_CLOUD_YAML_B64
  - OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR
  - OPENSTACK_DNS_NAMESERVERS
  - OPENSTACK_EXTERNAL_NETWORK_ID
  - OPENSTACK_FAILURE_DOMAIN
  - OPENSTACK_IMAGE_NAME
  - OPENSTACK_NODE_MACHINE_FLAVOR
  - OPENSTACK_SSH_KEY_NAME

Optional Variables:
  - CLUSTER_NAME                 (defaults to capi-quickstart)
  - CONTROL_PLANE_MACHINE_COUNT  (defaults to 1)
  - WORKER_MACHINE_COUNT         (defaults to 0)

Step 4 — Generate the Cluster Manifests

clusterctl generate cluster hind-cluster --infrastructure openstack:v0.14.1   --kubernetes-version v1.34.3  --control-plane-machine-count=1   --worker-machine-count=1  > capi-quickstart.yaml

The above command generates cluster manifests. Let's understand the generated manifests.

The Cluster manifest:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: hind-cluster
  namespace: default
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    serviceDomain: cluster.local
  controlPlaneRef:
    apiGroup: controlplane.cluster.x-k8s.io
    kind: KubeadmControlPlane
    name: hind-cluster-control-plane
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: OpenStackCluster
    name: hind-cluster

This is the main object — it ties everything together. It says:

Pods get IPs from 192.168.0.0/16
Control plane is defined by KubeadmControlPlane
OpenStack infrastructure is defined by OpenStackCluster

The OpenStackCluster - OpenStack Infrastructure Definition

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackCluster
metadata:
  name: hind-cluster
  namespace: default
spec:
  apiServerLoadBalancer:
    enabled: true
  externalNetwork:
    id: c926f5e2-830e-43b0-9d06-89641bb8e131
  identityRef:
    cloudName: openstack
    name: hind-cluster-cloud-config
  managedSecurityGroups:
    allNodesSecurityGroupRules:
    - description: Created by cluster-api-provider-openstack - BGP (calico)
      direction: ingress
      etherType: IPv4
      name: BGP (Calico)
      portRangeMax: 179
      portRangeMin: 179
      protocol: tcp
      remoteManagedGroups:
      - controlplane
      - worker
    - description: Created by cluster-api-provider-openstack - IP-in-IP (calico)
      direction: ingress
      etherType: IPv4
      name: IP-in-IP (calico)
      protocol: "4"
      remoteManagedGroups:
      - controlplane
      - worker
  managedSubnets:
  - cidr: 10.6.0.0/24
    dnsNameservers:
    - 8.8.8.8

CAPO controller in the capo-system namespace reads this manifest and:

Checks the existence of the external network using the ID provided in the above manifest, also creates an internal network for our cluster, subnet, a router, and attaches this internal network to the router so the nodes in the cluster get internet access (outbound) and a security group that allows only the load balancer to access the backend cluster nodes. The client should be able to access the cluster only through the loadbalancer IP and the backend control plane IPs, or their identities are not exposed.
Creates security groups for the control plane and workers.
Creates an Octavia LoadBalancer for the Kubernetes API endpoint. This is especially useful when u have multiple control plane nodes, and the load balancer will load balance the requests across the control planes.
Sets the LB IP as the cluster's controlPlaneEndpoint

The kubeadmControlPlane definition:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlane
metadata:
  name: hind-cluster-control-plane
  namespace: default
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      controllerManager:
        extraArgs:
        - name: cloud-provider
          value: external
    initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
        - name: cloud-provider
          value: external
        - name: provider-id
          value: openstack:///'{{ instance_id }}'
        name: '{{ local_hostname }}'
    joinConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
        - name: cloud-provider
          value: external
        - name: provider-id
          value: openstack:///'{{ instance_id }}'
        name: '{{ local_hostname }}'
  machineTemplate:
    spec:
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: OpenStackMachineTemplate
        name: hind-cluster-control-plane
  replicas: 1
  version: v1.34.3

cloud-provider: external on kubelet: When kubelet starts, it adds a taint node.cloudprovider.kubernetes.io/uninitialized on the node. No pods schedule here until the OpenStack Cloud Controller Manager (CCM) initializes the node and removes this taint. This flag tells kubelet, "don't try to handle cloud stuff yourself, wait for external CCM."

cloud-provider: external on controller-manager - Disables the built-in cloud loops in kube-controller-manager (node lifecycle, service LoadBalancer, routes). These are handed off to the OpenStack CCM pod. When a load-balancer type service is created, it is the responsibility of the CCM to create an LB for you behind the scenes.

provider-id: openstack:///{{ instance_id }} - A unique ID linking the Kubernetes Node object to the actual Nova VM. {{ instance_id }} is a cloud-init template variable resolved to the real VM UUID at boot time. The OpenStack CCM uses this to fetch instance metadata.

The OpenStackMachineTemplate for Control Plane:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: hind-cluster-md-0
  namespace: default
spec:
  template:
    spec:
      flavor: m1.medium
      image:
        filter:
          name: ubuntu-2404-kube
      sshKeyName: kashi

This tells CAPO what Nova flavor and Glance image to use for control plane VMs. The CAPO looks up the image in Glance and creates the VMs using the image UID fetched from Glance.

The MachineDeployment — Worker Node Pool

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: hind-cluster-md-0
  namespace: default
spec:
  clusterName: hind-cluster
  replicas: 1
  selector:
    matchLabels: null
  template:
    spec:
      bootstrap:
        configRef:
          apiGroup: bootstrap.cluster.x-k8s.io
          kind: KubeadmConfigTemplate
          name: hind-cluster-md-0
      clusterName: hind-cluster
      failureDomain: nova
      infrastructureRef:
        apiGroup: infrastructure.cluster.x-k8s.io
        kind: OpenStackMachineTemplate
        name: hind-cluster-md-0
      version: v1.34.3

Like a Deployment for VMs. It ensures the desired number of worker nodes exists. It references KubeadmConfigTemplate for the cloud-init script and OpenStackMachineTemplate for VM specs.

The KubeadmConfigTemplate — Worker Bootstrap Config

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: hind-cluster-md-0
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
          - name: cloud-provider
            value: external
          - name: provider-id
            value: openstack:///'{{ instance_id }}'
          name: '{{ local_hostname }}'

Workers only have joinConfiguration — they never run kubeadm init, only kubeadm join. The same cloud-provider: external and provider-id flags are set here for the same reasons as the control plane.

So, when you apply capi-quickstart.yaml, CAPI sees the cluster object, and CAPO sees the OpenstackCluster object and creates the internal network, subnet, router, attaches subnet to the router for internet access ( router will have a port from the provider network as external gw ). It also creates security groups and the loadbalancer for kubernetes API server endpoint.

You should see the below events once when you describe the OpenstackCluster resource:

Events:
  Type    Reason                         Age   From                  Message
  ----    ------                         ----  ----                  -------
  Normal  Successfulcreatenetwork        54m   openstack-controller  Created network k8s-clusterapi-cluster-default-hind-cluster with id 05371ddd-a37e-43d6-ba6d-58599c4d2b7b
  Normal  Successfulcreatesubnet         54m   openstack-controller  Created subnet k8s-clusterapi-cluster-default-hind-cluster with id ea8186bc-88f7-4cc6-aa27-bf9b5a1154c5
  Normal  Successfulcreaterouter         54m   openstack-controller  Created router k8s-clusterapi-cluster-default-hind-cluster with id 8790e4f0-3b55-4d4e-acf4-8ce37e5719f4
  Normal  Successfulcreatesecuritygroup  54m   openstack-controller  Created security group k8s-cluster-default-hind-cluster-secgroup-controlplane with id e08613ca-135c-4f64-ba67-def6a1ffd7e1
  Normal  Successfulcreatesecuritygroup  54m   openstack-controller  Created security group k8s-cluster-default-hind-cluster-secgroup-worker with id 4ed61b8a-a9b0-4fb8-a3db-af6e457aea8a
  Normal  Successfulcreateloadbalancer   54m   openstack-controller  Created load balancer k8s-clusterapi-cluster-default-hind-cluster-kubeapi with id 421cd8a9-a7ba-4f41-8f03-1cdc4d958358
  Normal  Successfulcreatefloatingip     52m   openstack-controller  Created floating IP 10.31.0.101 with id 433d5f18-34dc-489d-b2ae-19ba2a167f8b
  Normal  Successfulassociatefloatingip  52m   openstack-controller  Associated floating IP 10.31.0.101 with port 6f0f82d9-bc8e-42e1-88eb-65ec33469d6e
  Normal  Successfulcreatelistener       52m   openstack-controller  Created listener k8s-clusterapi-cluster-default-hind-cluster-kubeapi-6443 with id c66b31a9-f9ee-4bfd-b61f-b402c266a8bf
  Normal  Successfulcreatepool           52m   openstack-controller  Created pool k8s-clusterapi-cluster-default-hind-cluster-kubeapi-6443 with id 465f565c-3a84-4b4d-938b-c28dfcc009af
  Normal  Successfulcreatemonitor        52m   openstack-controller  Created monitor k8s-clusterapi-cluster-default-hind-cluster-kubeapi-6443 with id 1349252f-b42b-4393-bab3-9243ff9b282e

And Also verify the network, subnet, router, security groups in your openstack cluster:

(kolla-venv) root@control-1:~#  openstack network list
+--------------------------------------+---------------------------------------------+--------------------------------------+
| ID                                   | Name                                        | Subnets                              |
+--------------------------------------+---------------------------------------------+--------------------------------------+
| 05371ddd-a37e-43d6-ba6d-58599c4d2b7b | k8s-clusterapi-cluster-default-hind-cluster | ea8186bc-88f7-4cc6-aa27-bf9b5a1154c5 |
| 40c70b44-1c45-407f-b590-5c019814ef57 | trove-mgmt                                  | 22b30e5a-4cbb-422d-9e9b-6f0ee2736c46 |
| 75bba094-d69a-4060-a271-a5d529915f01 | private-net                                 | e2c43b43-2936-4839-b57c-8c955f44f913 |
| 7e72fa18-46b5-4bfb-96bf-5666df021dec | lb-mgmt-net                                 | 2183795b-8723-490f-aaa9-5bb49f2d9af1 |
| c926f5e2-830e-43b0-9d06-89641bb8e131 | provider                                    | 4073439c-176e-46b7-8f78-9e7debdfef02 |
+--------------------------------------+---------------------------------------------+--------------------------------------+
(kolla-venv) root@control-1:~#  openstack subnet list
 +--------------------------------------+---------------------------------------------+--------------------------------------+-----------------+
| ID                                   | Name                                        | Network                              | Subnet          |
+--------------------------------------+---------------------------------------------+--------------------------------------+-----------------+
| 2183795b-8723-490f-aaa9-5bb49f2d9af1 | lb-mgmt-subnet                              | 7e72fa18-46b5-4bfb-96bf-5666df021dec | 10.1.0.0/24     |
| 22b30e5a-4cbb-422d-9e9b-6f0ee2736c46 | trove-mgmt-subnet                           | 40c70b44-1c45-407f-b590-5c019814ef57 | 10.33.0.0/24    |
| 4073439c-176e-46b7-8f78-9e7debdfef02 | provider-subnet                             | c926f5e2-830e-43b0-9d06-89641bb8e131 | 10.31.0.0/24    |
| e2c43b43-2936-4839-b57c-8c955f44f913 | private-subnet                              | 75bba094-d69a-4060-a271-a5d529915f01 | 192.168.50.0/24 |
| ea8186bc-88f7-4cc6-aa27-bf9b5a1154c5 | k8s-clusterapi-cluster-default-hind-cluster | 05371ddd-a37e-43d6-ba6d-58599c4d2b7b | 10.6.0.0/24     |
+--------------------------------------+---------------------------------------------+--------------------------------------+-----------------+
(kolla-venv) root@control-1:~#  
(kolla-venv) root@control-1:~#  openstack router list 
+--------------------------------------+---------------------------------------------+--------+-------+----------------------------------+-------------+-------+
| ID                                   | Name                                        | Status | State | Project                          | Distributed | HA    |
+--------------------------------------+---------------------------------------------+--------+-------+----------------------------------+-------------+-------+
| 40625a37-d00e-422c-a5b4-577c4d4c2a81 | trove-mgmt-router                           | ACTIVE | UP    | 2c2343fd2bd141e8a18229afefb9c4ff | False       | False |
| 8790e4f0-3b55-4d4e-acf4-8ce37e5719f4 | k8s-clusterapi-cluster-default-hind-cluster | ACTIVE | UP    | 2c2343fd2bd141e8a18229afefb9c4ff | False       | False |
+--------------------------------------+---------------------------------------------+--------+-------+----------------------------------+-------------+-------+
(kolla-venv) root@control-1:~# 
(kolla-venv) root@control-1:~# 
(kolla-venv) root@control-1:~# 
(kolla-venv) root@control-1:~#  openstack router show k8s-clusterapi-cluster-default-hind-cluster
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                                                                     |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| admin_state_up          | UP                                                                                                                                                        |
| availability_zone_hints |                                                                                                                                                           |
| availability_zones      | nova                                                                                                                                                      |
| created_at              | 2026-03-29T11:26:23Z                                                                                                                                      |
| description             | Created by cluster-api-provider-openstack cluster default-hind-cluster                                                                                    |
| distributed             | False                                                                                                                                                     |
| enable_ndp_proxy        | None                                                                                                                                                      |
| external_gateway_info   | {"network_id": "c926f5e2-830e-43b0-9d06-89641bb8e131", "external_fixed_ips": [{"subnet_id": "4073439c-176e-46b7-8f78-9e7debdfef02", "ip_address":         |
|                         | "10.31.0.246"}], "enable_snat": true}                                                                                                                     |
| flavor_id               | None                                                                                                                                                      |
| ha                      | False                                                                                                                                                     |
| id                      | 8790e4f0-3b55-4d4e-acf4-8ce37e5719f4                                                                                                                      |
| interfaces_info         | [{"port_id": "1e65d210-c1e7-4118-a847-d27b9058077b", "ip_address": "10.6.0.1", "subnet_id": "ea8186bc-88f7-4cc6-aa27-bf9b5a1154c5"}]                      |
| name                    | k8s-clusterapi-cluster-default-hind-cluster                                                                                                               |
| project_id              | 2c2343fd2bd141e8a18229afefb9c4ff                                                                                                                          |
| revision_number         | 3                                                                                                                                                         |
| routes                  |                                                                                                                                                           |
| status                  | ACTIVE                                                                                                                                                    |
| tags                    |                                                                                                                                                           |
| updated_at              | 2026-03-29T11:26:25Z                                                                                                                                      |
+-------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
(kolla-venv) root@control-1:~# 
(kolla-venv) root@control-1:~# 
(kolla-venv) root@control-1:~# 
(kolla-venv) root@control-1:~#  openstack loadbalancer list
 +-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| id                                  | name                                | project_id                       | vip_address | provisioning_status | operating_status | provider |
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| 421cd8a9-a7ba-4f41-8f03-            | k8s-clusterapi-cluster-default-     | 2c2343fd2bd141e8a18229afefb9c4ff | 10.6.0.107  | ACTIVE              | ONLINE           | amphora  |
| 1cdc4d958358                        | hind-cluster-kubeapi                |                                  |             |                     |                  |          |
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
(kolla-venv) root@control-1:~#  
(kolla-venv) root@control-1:~#  openstack  security group list
+--------------------------------------+--------------------------------------------------------+---------------------------+----------------------------------+------+--------+
| ID                                   | Name                                                   | Description               | Project                          | Tags | Shared |
+--------------------------------------+--------------------------------------------------------+---------------------------+----------------------------------+------+--------+
| 188e18b6-cc7e-42d6-8b56-a03cb9af5e5a | lb-421cd8a9-a7ba-4f41-8f03-1cdc4d958358                |                           | f37717d8ea254fdfb12bff38e8834f9d | []   | False  |
| 266e525f-2e15-4f72-9c11-7448c47d9abf | trove-sec-grp                                          | trove-sec-grp             | 2c2343fd2bd141e8a18229afefb9c4ff | []   | False  |
| 3a9617cd-6e92-4c7f-bad5-fd3dfb1996ed | default                                                | Default security group    | 2c2343fd2bd141e8a18229afefb9c4ff | []   | False  |
| 429b9c68-e2db-437c-b4db-bf7f8bad6d60 | default                                                | Default security group    | f37717d8ea254fdfb12bff38e8834f9d | []   | False  |
| 4ed61b8a-a9b0-4fb8-a3db-af6e457aea8a | k8s-cluster-default-hind-cluster-secgroup-worker       | Cluster API managed group | 2c2343fd2bd141e8a18229afefb9c4ff | []   | False  |
| 64ad5172-e754-416b-a228-b1bd9d98e96a | test                                                   | test                      | 2c2343fd2bd141e8a18229afefb9c4ff | []   | False  |
| 80fbc123-07f5-4b7d-bc2c-92891fc96668 | lb-mgmt-sec-grp                                        |                           | f37717d8ea254fdfb12bff38e8834f9d | []   | False  |
| e08613ca-135c-4f64-ba67-def6a1ffd7e1 | k8s-cluster-default-hind-cluster-secgroup-controlplane | Cluster API managed group | 2c2343fd2bd141e8a18229afefb9c4ff | []   | False  |
| e8fc39aa-668a-404a-afe0-98ffa118f4a1 | lb-health-mgr-sec-grp                                  |                           | f37717d8ea254fdfb12bff38e8834f9d | []   | False  |
+--------------------------------------+--------------------------------------------------------+---------------------------+----------------------------------+------+--------+
(kolla-venv) root@control-1:~#

As you can see, the resources created by the clusterAPI are prefixed with k8s-clusterapi, and security groups are prefixed with k8s-cluster.

Also, you can see that a load balancer has been created that will act as the frontend for our control plane nodes.

Then the capi-kubeadm-control-plane-controller-manager will see the KubeadmControlPlane object and create the kubeadmconfig and machine objects in the cluster.

Verify:

kubectl get kubeadmconfig

Output:

NAME                               CLUSTER        DATA SECRET CREATED   AGE
hind-cluster-control-plane-wggsv   hind-cluster   true                  57m
hind-cluster-md-0-xg7gb-qjlvj      hind-cluster   true                  57m

We see 2 objects, one for controlplane and the other for the worker node.

kubectl get machine

Output:

NAME                               CLUSTER        NODE NAME                          READY     AVAILABLE   UP-TO-DATE   PHASE     AGE   VERSION
hind-cluster-control-plane-wggsv   hind-cluster   hind-cluster-control-plane-wggsv   Unknown   False       True         Running   57m   v1.34.3
hind-cluster-md-0-xg7gb-qjlvj      hind-cluster   hind-cluster-md-0-xg7gb-qjlvj      True      True        True         Running   57m   v1.34.3

And capi-kubeadm-bootstrap-controller-manager, does a crucial job. It generates the cloud-init script that contains the actual kubeadm commands to be run during the booting process of our nodes in the cluster. This cloud-init script is actually stored in a secret.

Highlighted above are the secrets created by the controller that contain the cloud-init script. These secrets will be referenced by the machine objects created by kubeadm-controlplane-mgr.

kubectl get machine hind-cluster-control-plane-wggsv -o yaml
apiVersion: cluster.x-k8s.io/v1beta2
kind: Machine
metadata:
  annotations:
    pre-terminate.delete.hook.machine.cluster.x-k8s.io/kcp-cleanup: ""
  creationTimestamp: "2026-03-29T11:28:04Z"
  finalizers:
  - machine.cluster.x-k8s.io
  generation: 3
  labels:
    cluster.x-k8s.io/cluster-name: hind-cluster
    cluster.x-k8s.io/control-plane: ""
    cluster.x-k8s.io/control-plane-name: hind-cluster-control-plane
  name: hind-cluster-control-plane-wggsv
  namespace: default
  ownerReferences:
  - apiVersion: controlplane.cluster.x-k8s.io/v1beta2
    blockOwnerDeletion: true
    controller: true
    kind: KubeadmControlPlane
    name: hind-cluster-control-plane
    uid: d25ecc23-dd2c-417a-b891-f88eb85225af
  resourceVersion: "15813"
  uid: 4b4725ad-f1ee-4232-8c06-4cd2f14ed025
spec:
  bootstrap:
    configRef:
      apiGroup: bootstrap.cluster.x-k8s.io
      kind: KubeadmConfig
      name: hind-cluster-control-plane-wggsv
    dataSecretName: hind-cluster-control-plane-wggsv
  clusterName: hind-cluster
  deletion:
    nodeDeletionTimeoutSeconds: 10
  failureDomain: nova
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: OpenStackMachine
    name: hind-cluster-control-plane-wggsv
  providerID: openstack:///3570638e-3d1c-4ccb-b758-86321356ea0d
  readinessGates:
  - conditionType: APIServerPodHealthy
  - conditionType: ControllerManagerPodHealthy
  - conditionType: SchedulerPodHealthy
  - conditionType: EtcdPodHealthy
  - conditionType: EtcdMemberHealthy
  version: v1.34.3

Verify the data secretName above. The value matches one of the secrets listed in the output of the get secrets command.

CAPO controller sees the machine object and calls the NOVA service to create the VM.

Also, while kubeadmcontrolplane is used for control plane nodes, machinedeployment is used for worker nodes.

After you applied the clusterAPI, manifests, get the kubeconfig of the workload cluster using:

clusterctl get kubeconfig hind-cluster > hind-cluster.kubeconfig

Export the kubeconfig and verify the cluster status:

export KUBECONFIG=./hind-cluster.kubeconfig
kubectl get nodes

Output:

NAME                               STATUS     ROLES           AGE   VERSION
hind-cluster-control-plane-wggsv   NotReady   control-plane   13m   v1.34.3
hind-cluster-md-0-xg7gb-qjlvj      NotReady   <none>          11m   v1.34.3

The nodes are in NotReady state. And it is completely expected. Because we need to install CNI.

You can install the CNI of your choice, I am going with calico:

kubectl --kubeconfig=./hind-cluster.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

After installing the CNI, nodes should transition to Ready state:

kubectl get nodes
NAME                               STATUS   ROLES           AGE   VERSION
hind-cluster-control-plane-wggsv   Ready    control-plane   70m   v1.34.3
hind-cluster-md-0-xg7gb-qjlvj      Ready    <none>          68m   v1.34.3

Also, remember that we discussed about the role of openstack cloud controller manager above. In the above capi manifests, especially for kubeadmcontrolplane under kubeadmconfigspec, and in the kubeadmconfigtemplate of machine deployment. we specified extra args. i.e cloud-provider set to external.

Since these flags are set, the worker node and control plane node both will register themselves with a taint: node.cluster.x-k8s.io/uninitialized:NoSchedule.

So unless you install Openstack cloud controller manager the nodes will stay unschedulable. Let's verify this by logging into the nodes.
Verify the kubelet status on control plane node:

On worker node:

You notice the following flags on both the nodes.
On worker node:

/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --pod-infra-container-image=registry.k8s.io/pause:3.10.1 --provider-id=openstack:///d9ae4dde-b281-421b-ad1b-36a2ee7db110 --register-with-taints=node.cluster.x-k8s.io/uninitialized:NoSchedule

The provider ID here corresponds to nova VM UUID. This is how OpenStack Cloud Controller Manager maps the Kubernetes node to nova vm object in OpenStack. Also, the cloud-provider is set to external.

On control plane:

usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --pod-infra-container-image=registry.k8s.io/pause:3.10.1 --provider-id=openstack:///3570638e-3d1c-4ccb-b758-86321356ea0d

Now check the taints.
For control plane:

For worker node:

So, now let's deploy OpenStack Cloud Controller Manager. Run the following steps:

wget https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-openstack/master/templates/env.rc -O /tmp/env.rc
chmod +x /tmp/env.rc
echo "Removing old cloud.conf files..."
rm -f /tmp/cloud.conf*
echo "Generating cloud.conf file from clouds.yaml..."
/tmp/env.rc ./clouds.yaml openstack
cp /tmp/cloud.conf* ./cloud.conf
echo "Creating secret in the kube-system ns..."
kubectl --kubeconfig=./hind-cluster.kubeconfig -n kube-system create secret generic cloud-config --from-file=cloud.conf
echo "Deploying cloud controller in the workload cluster..."
kubectl apply --kubeconfig=./hind-cluster.kubeconfig -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
kubectl apply --kubeconfig=./hind-cluster.kubeconfig -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
kubectl apply --kubeconfig=./hind-cluster.kubeconfig -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml

tmp.rc is the script used to generate cloud.conf file from clouds.yaml. Download the clouds.yaml file from your OpenStack cloud and specify the path to this script as the first argument. And the second argument is the cloud name in the clouds.yaml.

Once you install the cloud controller manager, it will remove the taint, and you should see the coredns pods move to the running state from pending.

Create a workload and Test the cluster:

Now lets create a small nginx pod and then expose it using Loadbalancer service. We will see if the openstack CCM creates an LB or not.

root@hind-cluster-control-plane-wggsv:~# kubectl create deploy nginx --image=nginx
deployment.apps/nginx created
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# kubectl get pods
NAME                     READY   STATUS              RESTARTS   AGE
nginx-66686b6766-549cc   0/1     ContainerCreating   0          4s
root@hind-cluster-control-plane-wggsv:~# kubectl expose deploy nginx --type=LoadBalancer --port=80
service/nginx exposed
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# 
root@hind-cluster-control-plane-wggsv:~# kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        90m
nginx        LoadBalancer   10.99.239.196   <pending>     80:31426/TCP   4s
root@hind-cluster-control-plane-wggsv:~#

Initially, it will be in the pending state. Describe the service and verify:

root@hind-cluster-control-plane-wggsv:~# kubectl describe svc nginx 
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              <none>
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.99.239.196
IPs:                      10.99.239.196
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31426/TCP
Endpoints:                192.168.69.193:80
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  12s   service-controller  Ensuring load balancer

Also verify in the openstack cluster:

(kolla-venv) root@control-1:~#  openstack loadbalancer list
  +-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| id                                  | name                                | project_id                       | vip_address | provisioning_status | operating_status | provider |
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| 421cd8a9-a7ba-4f41-8f03-            | k8s-clusterapi-cluster-default-     | 2c2343fd2bd141e8a18229afefb9c4ff | 10.6.0.107  | ACTIVE              | ONLINE           | amphora  |
| 1cdc4d958358                        | hind-cluster-kubeapi                |                                  |             |                     |                  |          |
| 5289ba7f-dc63-4db2-a6a4-            | kube_service_kubernetes_default_ngi | 2c2343fd2bd141e8a18229afefb9c4ff | 10.6.0.203  | PENDING_CREATE      | ONLINE           | amphora  |
| f72d2e86f33d                        | nx                                  |                                  |             |                     |                  |          |
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+

Finally after sometime, the status should change to active:

(kolla-venv) root@control-1:~#   openstack loadbalancer list
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| id                                  | name                                | project_id                       | vip_address | provisioning_status | operating_status | provider |
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| 421cd8a9-a7ba-4f41-8f03-            | k8s-clusterapi-cluster-default-     | 2c2343fd2bd141e8a18229afefb9c4ff | 10.6.0.107  | ACTIVE              | ONLINE           | amphora  |
| 1cdc4d958358                        | hind-cluster-kubeapi                |                                  |             |                     |                  |          |
| 5289ba7f-dc63-4db2-a6a4-            | kube_service_kubernetes_default_ngi | 2c2343fd2bd141e8a18229afefb9c4ff | 10.6.0.203  | ACTIVE              | ONLINE           | amphora  |
| f72d2e86f33d                        | nx                                  |                                  |             |                     |                  |          |
+-------------------------------------+-------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+

And the status of the service should change as follows:

root@hind-cluster-control-plane-wggsv:~# kubectl describe svc nginx 
Name:                     nginx
Namespace:                default
Labels:                   app=nginx
Annotations:              loadbalancer.openstack.org/load-balancer-address: 10.31.0.48
                          loadbalancer.openstack.org/load-balancer-id: 5289ba7f-dc63-4db2-a6a4-f72d2e86f33d
Selector:                 app=nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.99.239.196
IPs:                      10.99.239.196
LoadBalancer Ingress:     10.31.0.48 (VIP)
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31426/TCP
Endpoints:                192.168.69.193:80
Session Affinity:         None
External Traffic Policy:  Cluster
Internal Traffic Policy:  Cluster
Events:
  Type    Reason                Age                 From                Message
  ----    ------                ----                ----                -------
  Normal  EnsuringLoadBalancer  23s (x2 over 117s)  service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   23s (x2 over 23s)   service-controller  Ensured load balancer

root@hind-cluster-control-plane-wggsv:~# kubectl get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP      10.96.0.1       <none>        443/TCP        95m
nginx        LoadBalancer   10.99.239.196   10.31.0.48    80:31426/TCP   4m49s

you can see that the service got an IP from provider network. Let's verify it through curl:

root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, nginx is successfully installed and working.
Further configuration is required for the web server, reverse proxy, 
API gateway, load balancer, content cache, or other features.</p>

<p>For online documentation and support please refer to
<a href="https://nginx.org/">nginx.org</a>.<br/>
To engage with the community please visit
<a href="https://community.nginx.org/">community.nginx.org</a>.<br/>
For enterprise grade support, professional services, additional 
security features and capabilities please refer to
<a href="https://f5.com/nginx">f5.com/nginx</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@hind-cluster-control-plane-wggsv:~#

Now let's scale the pods to 2 replicas and see the loadbalancing in action.

root@hind-cluster-control-plane-wggsv:~# kubectl scale deploy nginx --replicas=2
deployment.apps/nginx scaled

# In the first replica:
root@nginx-66686b6766-549cc:/usr/share/nginx/html# echo "Replica-1" > index.html 
# In the second replica:
root@nginx-66686b6766-mgks5:/usr/share/nginx/html# echo "Replica-2" > index.html

Verify loadbalancing:

root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
Replica-2
root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
Replica-1
root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
Replica-1
root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
Replica-1
root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
Replica-1
root@hind-cluster-control-plane-wggsv:~# curl  10.31.0.48
Replica-2
root@hind-cluster-control-plane-wggsv:~#

That's all for now. If you found this helpful or got stuck somewhere, drop a comment below. This stuff took me a long time to piece together — happy to help you shortcut that journey.