Control Your Custom Cloud Resources with Pulumi

When I began backend development, I mainly worked with AWS, but recently I started looking into other cloud providers. Not the big ones like Azure or Google, but Auth0, Fauna, and Upstash.

While they don't offer as many services like AWS, they often bring attractive alternatives and a better developer experience. The problem is, not all of them are supported by IaC tools.

I'm using Pulumi to deploy and provision my cloud resources because it's provider-independent and allows me to use TypeScript, and in turn, JavaScript to define my infrastructure.

In this article, I will show you how to define your classes, so you can deploy whatever cloud resource you need. I will use Upstash as an example because deploying a serverless Redis database is probably as simple as it gets and still has practical relevance.

Note: The content here is based on a video by Cloudspeak. The main difference is that I used Upstash, a real cloud database as example. Also, I include the diff method and the video talks about the check and read methods, which I didn't cover here. Check it out!.

Prerequisites

You need an Pulumi and Upstash account. Also, the Pulumi CLI and Node.js installed.

The Goal

We want to write the following code, and the pulumi up command should lead to a serverless Redis database deployed with Upstash.

import * as upstash from "./upstash"

const redis = new upstash.Database("my-database", {
  region: "eu-west-1",
})

export const dbName = redis.name
export const dbEndpoint = redis.endpoint
export const dbPort = redis.port
export const dbPassword = redis.password

What We Need

Pulumi allows the implementation of custom resources in the Go programming language; these will then be compiled so that they're accessible from other languages, like TypeScript.

But it also has the concept of a dynamic resource, a custom resource implemented in TypeScript and only available in TypeScript.

A custom resource built with a dynamic Pulumi resource consists of two classes: The dynamic resource and a dynamic resource provider.

In the following code, I defined these two classes for our Redis database.

class Database extends pulumi.dynamic.Resource {
  ...
}

class DatabaseProvider implements pulumi.dynamic.ResourceProvider {
  ...
}

The Database uses the DatabaseProvider, and as we saw in the goal code example, the users only have to access the Database in their code to create a new instance of that resource.

The Database is used to define the inputs and outputs of our resource. So it doesn't know anything about the underlying service.

The DatabaseProvider handles all the interaction with the cloud service API to create, update, and delete the resource.

These two classes also run in different processes, so the only way to communicate between them are the methods the pulumi.dynamic.ResourceProvider interface defines.

Implementing the Database Class

Upstash provides a developer API, that allows managing databases programmatically. This API takes five inputs.

A name, that is a string.

A region, which currently only supports "eu-west-1", and "us-east-1".

A type, which can be "free" or "standard".

And an email and API key that needs to be supplied via HTTP basic authentication.

Our Database class will take these as inputs via its resource name and props in the constructor arguments. Since I'm using TypeScript here, I also tried to type the inputs correctly so we don't use invalid values.

The DatabaseInputs interface wraps all types in a pulumi.Input. This allows the Database class to accept the basic types directly or outputs from other resources. The provider will get the unwrapped outputs later.

The public attributes of the class are the outputs we can use later. In this case, we will use the inputs as outputs, but I also added some of the values that the API responds with after creation and update to connect to the database later.

type Region = "eu-west-1" | "us-east-1"
type Type = "free" | "standard"
type State = "active" | "deleted"

interface DatabaseInputs {
  name?: pulumi.Input<string>
  region: pulumi.Input<Region>
  type?: pulumi.Input<Type>
}

export class Database extends pulumi.dynamic.Resource {

  // Outputs from our inputs
  public readonly name!: pulumi.Output<string>
  public readonly region!: pulumi.Output<Region>
  public readonly type!: pulumi.Output<Type>

  // Outputs from API response
  public readonly port!: pulumi.Output<number>
  public readonly state!: pulumi.Output<State>
  public readonly password!: pulumi.Output<string>
  public readonly endpoint!: pulumi.Output<string>

  constructor(name: string, props: DatabaseInputs, opts?: any) {
    const config = new pulumi.Config("upstash")

    if (props.name === undefined) props.name = name
    if (props.type === undefined) props.type = "free"

    super(
      new DatabaseProvider(),
      name,
      {
        ...props,

        port: null,
        state: null,
        password: null,
        endpoint: null,

        upstashEmail: config.require("email"),
        upstashApiKey: config.require("apiKey"),
      },
      opts
    )
  }
}

The constructor takes three arguments.

A name, which is the resource name Pulumi will use. In this case, I re-used it for the database name in the props, because the Upstash API doesn't allow to change it right now. If we rename the resource, Pulumi will attempt to delete the database with the old name and create a database with the new name. So, that would be following the Upstash API anyway.

A props argument, which defines the configuration of our database. Currently, only three are available: name, region, type.

Note: You can only have one free database.

A opts argument, which defines Pulumi specific behavior for the resource. Like, protect it that it can't be accidentally deleted or with some parent resource. In this tutorial, that's not important, so I typed it any.

In the constructor, we gather the config values from the upstash namespace, so we don't have to add the email and API key as props, when creating the resource.

The calls to config.require expect a upstash:email and upstash:apiKey to be present in the Pulumi.dev.yaml file.

The email you used to create your Upstash account and the API key can be created in the Upstash console.

The values can be set with these CLI commands:

$ pulumi config set upstash:email <YOUR_EMAIL>
$ pulumi config set upstash:apiKey <YOUR_KEY> --secret

The constructor has to call super with an instance of the DatabaseProvider, the resource name, the props, and the opts.

The props also have to include null-ish values for all the outputs we will later get from the server and the values we got from the config. They will be available as arguments for the methods of the DatabaseProvider later.

It's crucial to remember the order of fields in the third object passed to super. These will be serialized and given to the provider class in another process. If the fields have a different order later, this will lead to an error where the resource is deployed, but Pulumi thinks the deployment failed.

Note: The two classes run in different processes, so the props are the only way to give the provider data. Calls to pulumi.Config aren't possible in the provider either.

The Database class knows nothing about Upstash, it's only responsible for telling Pulumi there is a new resource, and it has specific inputs and outputs. This way, we can interact with our resources as we would with all the other resources provided by Pulumi and seen in the goal section initially.

Implementing the DatabaseProvider Class

The next step is to implement the DatabaseProvider, which will do the actual work.

The DatabaseProvider has to implement the pulumi.dynamic.ResourceProvider interface, which only requires a create method to be implemented.

For our database, we will need four methods: create, update, delete, and diff.

The first time we run pulumi up with our Database resource, the create method will be called.

After that, the diff method will be called to check if anything of importance changed and if it can be applied to our resource via an update call or if it requires us to replace the resource, which essentially means a call to create and delete or delete and create.

Let's go over every method in detail, starting with create. I also provided the types again, so we know what is what.

Note: The types aren't wrapped in pulumi.Input anymore.

type Region = "eu-west-1" | "us-east-1"
type Type = "free" | "standard"
type State = "active" | "deleted"

interface DatabaseProviderInputs {
  name: string
  region: Region
  type: Type

  port: number
  state: State
  password: string
  endpoint: string

  upstashEmail: string
  upstashApiKey: string
}

...

async create(inputs: DatabaseProviderInputs) {
    const { data } = await axios.post(
      "https://api.upstash.com/v1/database",
      { database_name: inputs.name, region: inputs.region, type: inputs.type },
      {
        auth: {
          username: inputs.upstashEmail,
          password: inputs.upstashApiKey,
        },
      }
    )

    return {
      id: data.database_id,
      outs: {
        ...inputs,

        port: data.port,
        state: data.state,
        password: data.password,
        endpoint: data.endpoint,

        upstashEmail: inputs.upstashEmail,
        upstashApiKey: inputs.upstashApiKey,
      },
    }
  }

  ...

The method is asynchronous, which means it returns a promise. This allows us to access the network or the disk to deploy our database resource.

In this case, we need to call the Upstash developer API via HTTP. I'm using the axios package to make things a bit more convenient.

The inputs argument holds all data we need to call the API. It's a POST request with the name, region, and type in the body and the API credentials supplied via HTTP basic authentication.

If everything went well, we could use the data attribute of our result to return the new state of our database resource to Pulumi.

The database_id becomes our Pulumi ID, and the rest will be put in the outs object. This object needs the same field order as the object in the super call before. Otherwise, we get an error even if the deployment worked.

The axios client will throw an error when it didn't receive a 200 HTTP status. This will automatically be picked up by Pulumi and displayed to the user.

Next in line is the delete method. Which will be called when our database has to go.

  async delete(id: string, inputs: DatabaseProviderInputs) {
    await axios.delete(`https://api.upstash.com/v1/database/${id}`, {
      auth: {
        username: inputs.upstashEmail,
        password: inputs.upstashApiKey,
      },
    })
  }

We use the first argument with a DELETE request, but depending on the API you use, it could be that you need other values. That's why the method also has access to the inputs again.

The diff method is used for updates.

There are two types of updates—regular updates and replacements.

A regular update will use the existing database and change its configuration.

A replacement will delete the database and redeploy it with a new configuration. We can decide if we delete the old database first and then create a new one or if we want to make the new one first and then delete the old one. In our Upstash use-case is essential to delete first because we can only have one free database.

If we didn't implement the update method, a replacement is the only available option. But let's not get ahead of us and look at the diff method first.

  async diff(
    id: string,
    olds: DatabaseProviderInputs,
    news: DatabaseProviderInputs
  ) {
    const diffResult: any = { changes: false }

    const replaces = []
    if (olds.name !== news.name) replaces.push("name")
    if (olds.region !== news.region) replaces.push("region")
    if (replaces.length > 0) diffResult.replaces = replaces

    if (
      replaces.length > 0 ||
      olds.upstashEmail !== news.upstashEmail ||
      olds.upstashApiKey !== news.upstashApiKey
    ) {
      diffResult.changes = true
      diffResult.deleteBeforeReplace = true
    }

    return diffResult
  }

The method has to return a diffResult which contains a changes boolean field that tells Pulumi something has changed.

The diffResult can also contain a replaces string array field. If this field is present, Pulumi will go for a replacement. If replaces is missing, Pulumi will try a regular update.

The replaces field is a list of all inputs that were responsible for the replacement. In the Upstash case, only the type can be upgraded from "free" to "standard". A name and region change requires a replacement.

Pulumi will inform the user about the inputs that caused the replacement before the change will be executed, so they don't delete their database by accident.

The deleteBeforeReplace field will change the order of creating and delete, so the database is gone before creating a new one.

The last method is update, which will try to change the database configuration in place.

  async update(
    id: string,
    olds: DatabaseProviderInputs,
    news: DatabaseProviderInputs
  ) {
    if (news.type !== olds.type) {
      await axios.post(
        `https://api.upstash.com/v1/database/${id}/upgrade`,
        { type: news.type },
        {
          auth: {
            username: news.upstashEmail,
            password: news.upstashApiKey,
          },
        }
      )
    }
    return {
      outs: {
        ...news,

        port: olds.port,
        state: olds.state,
        password: olds.password,
        endpoint: olds.endpoint,

        upstashEmail: news.upstashEmail,
        upstashApiKey: news.upstashApiKey,
      },
    }
  }

The method gets the old inputs and the new inputs, and the Pulumi ID of the resource. There is nothing special happening here, just an API call to the upgrade endpoint if the type changed.

Again, the outs need to be in the correct order; otherwise: fails that lead to inconsistencies.

Conclusion

Dynamic resources are tricky in terms of input field ordering and wrapping types correctly, but overall they are pretty flexible.

If you can define a resource or a task that needs to be done at deployment in terms of create, update, delete, and diff it can be implemented as a dynamic resource.

I think TypeScript makes things incredibly convenient. We can define many of the inputs as static types that will increase developer experience when using the resource and notify us that things are wrong before we even start deploying.

Comments

[Tobias Schmidt]

Thank you for this great article.
Helped me a lot to implement this for my own use-case 🎉

[K]

You are welcome! :)