Building a Cloud Development Kit (CDK)

What Exactly is a Cloud Development Kit (CDK)?

Imagine you're a developer who needs to set up a bunch of cloud resources. Traditionally, you might deal with endless lines of JSON or YAML configurations. It's precise but can get pretty tedious, right? Well, this is where a Cloud Development Kit, or CDK, comes in handy. Instead of those endless configuration files, you use a programming language you’re already comfortable with—like TypeScript, Python, or Java. This means you can code your cloud infrastructure just like you’d code anything else.

Why is the CDK a Game Changer?

Let’s break it down:

1. Productivity on Steroids: Forget about switching gears between applications and infrastructure. Now, it's all in one place, with tools you already love (and understand!). Autocomplete, refactoring, and error checking are right there with you.

2. Reuse and Recycle: Craft a piece of infrastructure once, wrap it up into a component, and reuse it anywhere you need it. This not only saves time but also keeps your setups consistent.

3. Tailor-Made Solutions: Extend the basic setups with your own tweaks. Need a special kind of storage or a unique authentication method? Just code it in.

4. Transparent and Controlled: It’s all in your codebase, visible and version-controlled. Every change is clear and trackable—no surprises.

AWS CDK

AWS was one of the first to jump on this bandwagon with their CDK. It takes the power of AWS CloudFormation and makes it friendlier. Instead of wading through those YAML or JSON templates, you write in a comfortable, expressive programming language.

Here’s an example of AWS CDK with TypeScript:

import * as cdk from '@aws-cdk/core';
import * as s3 from '@aws-cdk/aws-s3';

class MyCloudStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Setting up an S3 bucket with version control
    new s3.Bucket(this, 'MyFirstBucket', {
      versioned: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });
  }
}

const app = new cdk.App();
new MyCloudStack(app, 'MyCloudStack');

See how intuitive and straightforward defining cloud resources can be with CDK. It’s just like writing any other piece of software!

AWS CDK Workflow Explained

The AWS Cloud Development Kit (AWS CDK) utilizes a sophisticated workflow that integrates various AWS services to provide a seamless development and deployment experience. Here’s a breakdown of the typical workflow and how it leverages these services:

1. Define Infrastructure as Code: You start by defining your cloud resources using familiar programming languages such as TypeScript or Python. This code defines what resources you need, such as databases, storage buckets, or compute instances.

2. Synthesis: The AWS CDK app takes this high-level code and compiles it into a lower-level CloudFormation template. This process, known as synthesis, involves the CDK CLI (cdk synth) transforming your declarative setup into a set of instructions that AWS CloudFormation can understand.

3. Deployment: Once the CloudFormation template is synthesized, you deploy it using the CDK CLI (cdk deploy). This command instructs AWS CloudFormation to provision and manage the resources as specified in the template.

4. Utilization of AWS Services: Throughout this process, AWS CDK interacts with various AWS services like AWS CodePipeline and AWS CodeCommit for continuous integration and delivery, enhancing the CI/CD pipeline's efficiency and robustness.

Building a CDK

Now that we understand what a CDK is, let's look at designing one for a cloud platform. Here, I'll be focusing on tau, an open-source CDN PaaS I founded.

Tau uses YAML for resource definition. This approach is not unique; Kubernetes and many other platforms use YAML as well. However, what sets tau apart is its design philosophy: Git is the single source of truth. This means there are no APIs to call for defining resources like storage buckets. Instead, everything is managed through Git.

This design is advantageous because it essentially provides us with an equivalent to AWS CloudFormation, but integrated directly with Git. This tight integration simplifies infrastructure management and makes it inherently version-controlled and collaborative.

All we need to do is figure out a way to generate and edit YAML files with code.

The Schema Package

Because Tau is a cloud platform you can write tests for, we already built a package called schema to manipulate YAML configuration files. The catch is that it has to be in Go, while most CDKs at least support JavaScript/TypeScript and Python.

This creates a challenge: how can we extend our CDK to support these popular languages while leveraging the existing schema package?

WebAssembly

Go is one of the first languages that supported WebAssembly as a compilation target. Additionally, most programming languages either have a native implementation or bindings for a WebAssembly runtime.

So, if we compile the schema package into a WebAssembly module, we should be able to call it from a variety of languages. This approach leverages the strengths of Go and the flexibility of WebAssembly, making it possible to interact with Tau's infrastructure definitions from languages like JavaScript, TypeScript, and Python.

Extism

Writing code to initiate a WebAssembly runtime, load our module, and call exported functions can be quite a lot of work. Luckily, the team at dylibso has thought about that and built Extism, a framework for building plugins with WebAssembly.

Example of Extism Plugin

Here’s a simple example of a Go plugin using Extism:

package main

import (
    "github.com/extism/go-pdk"
)

//export greet
func greet() int32 {
    input := pdk.Input()
    greeting := `Hello, ` + string(input) + `!`
    pdk.OutputString(greeting)
    return 0
}

func main() {}

Loading and calling greet from JavaScript is straightforward:

import createPlugin from '@extism/extism';

const plugin = await createPlugin('plugin.wasm', {
  useWasi: true,
});

const out = await plugin.call('greet', 'Samy');
console.log(out.text());

await plugin.close();

Using Extism, we can easily compile our Go code into WebAssembly and call it from other languages like JavaScript, making the integration process seamless and efficient. This capability significantly reduces the complexity of working with WebAssembly, enabling us to extend Tau’s CDK functionalities to various programming environments effortlessly.

The Plugin

The plugin is a wrapper around the schema package that uses the Extism PDK. Just like schema, the first step involves opening a project by reading a folder that contains configuration files (the folder can also be empty).

Here's how you can do it in Go:

package main

import (
    "github.com/extism/go-pdk"
    proj "github.com/taubyte/tau/pkg/schema/project"
)

var project proj.Project

//export openProject
func openProject() int32 {
    var err error
    project, err = proj.Open(proj.SystemFS("/mnt"))
    if err != nil {
        pdk.SetError(err)
        return 1
    }

    return 0
}

You might notice the global variable project. This isn't a novice mistake:

1. We will load the module each time we open a project.
2. WebAssembly is single-threaded, at least for now.

The JavaScript CDK

On the JavaScript side, we have:

import createPlugin from '@extism/extism';
import * as path from 'path';

export async function core(mountPath: string): Promise<any> {
    const coreWasmPath = path.resolve(__dirname, '../core.wasm');

    return await createPlugin(
        coreWasmPath,
        {
            useWasi: true,
            allowedPaths: { '/mnt': mountPath },
        }
    );
}

This function, core, loads the module and attaches the folder containing the configuration files to it as /mnt.

To open a project, we load the Wasm module, call openProject, and then return a Project object that references an instance of the core module initialized to our project.

export async function open(mountPath: string): Promise<Project> {
    const plugin = await core(mountPath);

    await plugin.call('openProject');

    return new Project(plugin);
}

From there, we can write code like this:

import open from '@taubyte/cdk';

const prj = await open('path/to/folder');
await prj.functions().new({
  name: 'ping',
  method: 'http',
  // other configurations
});

await prj.close();

Next Steps

Now that we have a CDK, the next step is to integrate it with Tau's CI/CD workflow. This integration will enable seamless deployment and management of cloud resources directly from your development pipeline. I’ll cover this integration in a separate article, where we will dive into the specifics of automating your deployments.

Conclusion

In this article, we explored what a Cloud Development Kit (CDK) is and why it’s a game changer for cloud infrastructure management. We looked at how AWS CDK simplifies defining and deploying cloud resources using familiar programming languages.

By leveraging WebAssembly and the Extism framework, we were able to create a versatile CDK that supports multiple languages, making tau’s infrastructure management more accessible and efficient.

My vision, is that this will evolve to also allows developers to embed infrastructure definitions directly within their application code, providing a seamless and cohesive development experience.

Stay tuned for the next article, where we’ll dive into integrating the CDK with tau’s CI/CD workflow.