Building Custom Control Planes using Crossplane

It's been about a year when we wrote about Crossplane. The previous post explains how Crossplane helps you provision and manage infrastructure using the Kubernetes API. Crossplane has since evolved and reached v1.0! Big congratulations to the Crossplane community!
Besides being an interface to cloud providers, Crossplane now has mechanisms to create and publish your own custom resources, without writing code. It offers ways to define your own infrastructure resources declaratively, in form of CRDs on top of cloud providers APIs. In this post, we'll see how you can use Crossplane to create your own control plane on top of cloud providers.

Infrastructure Provisioning

Let's take a look at how Crossplane allows us to provision cloud resources.

Installation
The below commands will install Crossplane in a Kubernetes cluster using Helm 3

$ kubectl create namespace crossplane-system

$ helm repo add crossplane-stable https://charts.crossplane.io/stable
$ helm repo update

$ helm install crossplane --namespace crossplane-system crossplane-stable/crossplane

Crossplane CLI is kubectl plugin to manage Crossplane packages. To install, run:
curl -sL https://raw.githubusercontent.com/crossplane/crossplane/release-1.0/install.sh | sh

Working with a Cloud Provider
A "provider" in Crossplane is a bunch of CRDs and their controllers, creating an interface to a cloud provider API (in fact, any API). We'll use AWS for demonstration here.

Install the provider
kubectl crossplane install provider crossplane/provider-aws:v0.16.0

Create a provider secret from the AWS credentials by running the following. This uses the default profile from AWS CLI credentials.

$ AWS_PROFILE=default && echo -e "[default]\naws_access_key_id = $(aws configure get aws_access_key_id --profile $AWS_PROFILE)\naws_secret_access_key = $(aws configure get aws_secret_access_key --profile $AWS_PROFILE)" > creds.conf

$ kubectl create secret generic aws-creds -n crossplane-system --from-file=key=./creds.conf

And a ProviderConfig to configure credentials for the AWS provider:

$ cat > providerconfig.yaml <<EOF
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: key
EOF

$ kubectl apply -f providerconfig.yaml

Below is a Crossplane representation of a AWS RDS instance, called a "managed resource". A Crossplane managed resource is a representation of a cloud provider resource and in fact, any API.

$ cat > rds.yaml <<EOF
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: rdspostgresql
spec:
  forProvider:
    region: us-east-1
    dbInstanceClass: db.t2.small
    masterUsername: masteruser
    allocatedStorage: 20
    engine: postgres
    engineVersion: "9.6"
    skipFinalSnapshotBeforeDeletion: true
  writeConnectionSecretToRef:
    namespace: crossplane-system
    name: aws-rdspostgresql-conn
EOF

$ kubectl apply -f rds.yaml

This will create an RDS instance and a Kubernetes Secret with the credentials for it.

Infrastructure Composition

While managed resources are great for managing cloud resources, it can soon become overwhelming to manage lots of them each time. For instance, an application developer might not be interested in the details of how a RDS instance is being created and managed with firewall rules, database configuration, etc. The only thing of interest might be a database and credentials to access it. On the contrary, an infrastructure operator may not want to expose all of the RDS configurations to a developer, but instead restrict only to some necessary parameters.

Crossplane offers mechanisms to compose managed resources where users can create their own abstractions in a declarative way.
Composite Resources: A Composite Resource (XR) is a custom resource that is composed of managed resources allowing you to abstract the infrastructure details.
A CompositeResourceDefinition (XRD) defines a new kind of composite resource. XRDs are cluster scoped. To create a namespaced XR, the respective XRD may optionally offer a Composite Resource Claim (XRC).

Composition: A Composition specifies which resources a XR will be composed of i.e what happens when you create XR. There can be multiple compositions for one XR. For example, for a CompositeDatabase XR, you may have a composition which will create a AWS RDS instance, a security group and a MySQL database. Another composition can define GCP CloudSQL instance and a PostgreSQL database.

Configuration: A configuration is a package of XRDs and Compositions that can be then published using the Crossplane CLI to Crossplane by creating a declarative Configuration resource

The below figure shows how the Crossplane resources might look like in a Kubernetes cluster. The XRDs are cluster scoped with compositions satisfying them. These XRDs offer a XRC each, which are namespace scoped. Offering a XRC is optional and is done to allow users to create XRs in their own namespace.
The XRDs also are referring to a managed resource in the cluster scope, so that the XRC can use it.

Let's see a quick example to understand these concepts better. We'll create a CompositePostgreSQLInstance XRD for PostgreSQL database , a composition for satisfying this XRD, and see how we can use the XRC that gets created.

You can leverage the Crossplane CLI to install a configuration package:
kubectl crossplane install configuration registry.upbound.io/xp/getting-started-with-aws:v1.0.0

Otherwise, you can define the XRD and composition yourself as explained below.
(The examples are taken from official Crossplane docs)

The XRD is defined below. Note how a claim is offered under claimNames. The only configurable field in the spec is storageGB

// xrd.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: compositepostgresqlinstances.database.example.org
spec:
  group: database.example.org
  names:
    kind: CompositePostgreSQLInstance
    plural: compositepostgresqlinstances
  claimNames:
    kind: PostgreSQLInstance
    plural: postgresqlinstances
  connectionSecretKeys:
    - username
    - password
    - endpoint
    - port
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  storageGB:
                    type: integer
                required:
                  - storageGB
            required:
              - parameters

The composition defined below defines creation of a AWS RDS instance. Notice how it references the XRD using the compositeTypeRef field. The resources define a set of base resources that get created when this composition is used. And under that, the patches field is used to fetch the property values from the XRD.

// composition.yaml
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: compositepostgresqlinstances.aws.database.example.org
  labels:
    provider: aws
    guide: quickstart
    vpc: default
spec:
  writeConnectionSecretsToNamespace: crossplane-system
  compositeTypeRef:
    apiVersion: database.example.org/v1alpha1
    kind: CompositePostgreSQLInstance
  resources:
    - base:
        apiVersion: database.aws.crossplane.io/v1beta1
        kind: RDSInstance
        spec:
          forProvider:
            region: us-east-1
            dbInstanceClass: db.t2.small
            masterUsername: masteruser
            engine: postgres
            engineVersion: "9.6"
            skipFinalSnapshotBeforeDeletion: true
            publiclyAccessible: true
          writeConnectionSecretToRef:
            namespace: crossplane-system
      patches:
        - fromFieldPath: "metadata.uid"
          toFieldPath: "spec.writeConnectionSecretToRef.name"
          transforms:
            - type: string
              string:
                fmt: "%s-postgresql"
        - fromFieldPath: "spec.parameters.storageGB"
          toFieldPath: "spec.forProvider.allocatedStorage"
      connectionDetails:
        - fromConnectionSecretKey: username
        - fromConnectionSecretKey: password
        - fromConnectionSecretKey: endpoint
        - fromConnectionSecretKey: port

Install these two manifests

$ kubectl apply -f xrd.yaml && kubectl apply -f composition.yaml

You can now see a new XRC created, postgresqlinstances.database.example.org. Use this XRC to create a AWS RDS instance.

// xr.yaml
apiVersion: database.example.org/v1alpha1
kind: PostgreSQLInstance
metadata:
  name: my-db
  namespace: default
spec:
  parameters:
    storageGB: 20
  compositionSelector:
    matchLabels:
      provider: aws
      vpc: default
  writeConnectionSecretToRef:
    name: db-conn

$ kubectl apply -f xr.yaml

This will result in creation of RDS PostgreSQL instance. Similarly, you can have another Composition for GCP CloudSQL and create a XR by modifying the compositionSelector field in the above manifest.

Creating Control Planes

As we saw, Crossplane helps to build custom resources on top of cloud resources and in fact on top of any API. This is great to build a control plane with required abstractions and restrictions using XRD, composition and XRC.

The below depiction shows an example of how an infrastructure operator might allow other people to create their database instances which are attached to VPCs which they don't have control of.

The image shows a VPC at cluster scope. Also at the cluster level are two database XRDs offering a claim each. Other users can then create their own database XRs using the namespace scoped XRC and still refer to the VPC. There can be multiple XRDs with different configurations on the cluster scope, allowing users to choose from. Also, different compositions for different cloud providers to satisfy those XRDs.

The defined resource definitions can then be packaged and distributed as a configuration. You can read more how the Crossplane CLI helps you do this.

Conclusion

Crossplane helps to provision infrastructure using the Kubernetes API. Moreover, it helps to build custom abstractions and restrictions over any API. This makes it a great tool to build a single control plane to manage multiple cloud providers.

We hope this post gives you a good start for using Crossplane. We're always thrilled to connect to people working with cloud native technologies. You can reach out to us via Twitter and LinkedIn.