Keep Application Pipelines Simple by Provisioning Managed Cloud Services using Kubernetes YAML

Teams adopt Kubernetes because the fully declarative Kubernetes YAML can drastically reduce the complexity of application deployment automation. This is also a key enabler for GitOps.

However, applications that depend on managed cloud services like e.g. databases, object storage buckets or queues require the deployment automation to also handle these non Kubernetes resources. Since this is a common use-case, the full benefit Kubernetes brings to application deployment is often unfortunately drastically reduced.

Teams with this requirement have three options to move forward:

1. Handle the infrastructure dependencies in a separate pipeline.
2. Handle the infrastructure dependencies in the application pipeline itself.
3. Extend Kubernetes to provide a declarative way to handle these infrastructure dependencies as well.

Options 1 and 2 are well known but have significant downsides. Handling these non Kubernetes infrastructure dependencies in a separate pipeline requires manual orchestration for changes that affect both infrastructure services and the application. But implementing this orchestration in a single pipeline is anything but trivial. For teams choosing options 1 or 2, Terraform is a popular choice because it can handle both infrastructure resources and Kubernetes resources, but the Terraform integration with Kubernetes also has some limitations to be aware of.

With Kubestack, the open-source Terraform GitOps Framework I maintain, I strongly encourage teams to keep infrastructure and application automation strictly separated from each other because interdependencies are a frequent source of blockers between different tasks. So while Kubestack uses Terraform to provision the AKS, EKS or GKE clusters and all cluster services that are required before applications can be deployed, using the same pipeline to provision the managed database service an application needs is discouraged because it would break the important separation.

This leaves us with option 3. In principle, the idea is to use custom resources, to extend the Kubernetes API to support additional resource types, and then have custom controllers, commonly called operators, manage the lifecycle of these new resource types the same way a built-in controller handles deployment resources. This keeps the logic out of the application pipelines giving us back the simplicity of handling just the Kubernetes YAML.

While there are also operators that run the service workloads themselves inside the Kubernetes clusters, I will only be looking at solutions that help provision managed cloud services for this post.

Let's take a look at some available projects:

AWS Service Operator

At first glance the AWS Service Operator does not look like a project anyone should trust their infrastructure automation with.

The first version was a thin wrapper around cloud formation templates and has since been archived. The blog post that announced the original version end of 2018 has been edited in February 2020 to refer to an effort to rewrite the operator that started mid-2019. Looking at the MVP branch in the new repository shows recent activity and gives the impression of active development. But there hasn’t been a release yet.

Still, the possibility to have an operator that allows managing AWS resources from within Kubernetes using custom resources built and maintained by AWS engineers justifies keeping an eye on the AWS Service Operator for teams running on AWS.

Update Aug, 20 as of today, AWS has released a preview of the new version under a new name. Read more about the new AWS Controller for Kubernetes on the AWS blog.

Azure Service Operator

A few days ago, Azure announced the release of the Azure Service Operator. The repository shows active development and the latest release was 10 days ago. This leaves a solid first impression.

The Azure Service Operator states support for a number of Azure services like EventHub, Azure SQL, CosmosDB, Storage Accounts, and more. For teams running applications on Kubernetes on Azure, having access to an operator maintained by Microsoft is a promising option.

Crossplane

Where the AWS and Azure Service Operators are cloud specific, Crossplane is an open-source project that offers a multi-cloud solution. The repository shows active and recent development. Crossplane is divided into a cloud agnostic part and cloud specific providers. These providers exist for AWS, GCP, Azure and Alibaba.

Crossplane is supported by Upbound and not one of the cloud providers directly, but makes up for that by offering multi-cloud support.

Even if multi-cloud support is not a requirement, Crossplane is worth taking a look at because it has been around significantly longer which suggests more maturity. Looking at the custom resources Crossplane makes available inside the cluster it seems there are about 70 different services supported between the four cloud providers.

Terraform Operator

Hashicorp in March announced the alpha release of the official Terraform operator. Terraform already has wide support for cloud resources across many cloud providers including their managed services. But as discussed earlier, having to maintain both Kubernetes YAML and Terraform HCL in one repository and having to run Terraform as part of the application pipeline adds complexity to that pipeline.

The Terraform operator takes the approach that users specify a Workspace custom resource which it syncs with the workspace on the mandatory linked Terraform cloud account. Based on the activity in the repository, the Terraform operator also seems actively developed.

For teams already invested into Terraform, this may be a good option. Given Terraform’s wide support, it may also support significantly more services than the other three. But this approach isn’t as purpose built as the other three options we discussed earlier and may require all team members to get comfortable with HCL and the Terraform concepts of providers and modules.

Conclusion

For teams that want to keep their application pipelines simple and only have to worry about maintaining Kubernetes YAML to specify their application’s requirements, there are options. Handling it this way makes perfect sense, because it helps further strengthening the separation between application and platform layer that Kubernetes enables.

With the AWS and Azure Service Operators there are two examples of implementations directly from the cloud providers. Crossplane is a multi-cloud alternative maintained as an open-source project. And with the Terraform Operator, there is also an option to keep the application pipeline simple but still having access to the Terraform ecosystem.

None of the solutions have announced general availability yet so it’s too early to make any recommendations. But the approach described here is powerful to help teams clearly separating between application and platform automation and this very much justifies keeping an eye on the future development in this space.