DEV Community: Adrien F

Use Cloudflare Workers to store your Terraform states

Adrien F — Tue, 09 Jan 2024 20:46:04 +0000

TL;DR: Check out the resulting worker on Github

When starting a Cloudflare focused project, you’re probably tempted like me to use Terraform to configure everything the right way, and quickly comes the question of where to store the states.

Terraform Cloud offers a free tier but is limited to 500 managed resources across all states (from what I understand of the pricing page), and if you're really starting from scratch, you might not already have one of the supported backends ready, be it an object storage engine, Kubernetes or PostgreSQL. So what are we left with?

Well, for a few years now, Cloudflare has offered an object storage engine called R2! And what’s even better, is that it’s compatible with the S3 API, enabling us to use it with the Terraform S3 backend. Will it be that easy?

The non-obvious S3 way

You might be thinking that since Cloudflare implemented the S3 API, it should be quite easy to configure Terraform to use it.

Create a new R2 bucket in the Cloudflare dashboard, and then a new API token scoped to the bucket. You will be provided with the proper credentials and URLs to provide to Terraform:

Finally, we just need to override the S3 URL with our bucket, provide our access keys and voilà, we can run terraform init and go on our way. You start to read the documentation and see that you can indeed override the endpoints. It might look like this:



terraform {
  backend "s3" {
    bucket     = "tf-states"
    key        = "foo.tfstate"
    endpoints  = { s3 = "https://xxx.r2.cloudflarestorage.com" }
    access_key = "xxx"
    secret_key = "xxx"
  }
}

⚠ Please don't commit your access/secret keys, use environment variables or a secret engine 😅

But sadly, you will quickly get an error:



❯ terraform init

Initializing the backend...
╷
│ Error: Missing region value
│
│   on main.tf line 2, in terraform:
│    2:   backend "s3" {
│
│ The "region" attribute or the "AWS_REGION" or "AWS_DEFAULT_REGION" environment variables must be set.

Alright, that makes sense, let’s add a region parameter. Reading R2 documentation, we can safely set it to us-east-1 .

This will look like this, will we finally get lucky?



terraform {
  backend "s3" {
    bucket     = "tf-states"
    key        = "foo.tfstate"
    endpoints  = { s3 = "https://xxx.r2.cloudflarestorage.com" }
    access_key = "xxx"
    secret_key = "xxx"
    region = "us-east-1"
  }
}

Wrong!



❯ terraform init

Initializing the backend...
╷
│ Error: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: abb0da81-771e-4724-ad9b-842ceb81e6de, api error InvalidClientTokenId: The security token included in the request is invalid.
│
│
╵

And so this will go on and on. This Github issue summarizes everything everyone went through to identify the right set of parameters, and you will also find more information if you want to use the AWS provider to use some S3 resources with R2. This is how your backend configuration should look like:



terraform {
  backend "s3" {
    bucket     = "tf-stats"
    key        = "foo.tfstate"
    endpoints  = { s3 = "https://xxx.r2.cloudflarestorage.com" }
    access_key = "xxx"
    secret_key = "xxx"
    region     = "us-east-1"

    skip_credentials_validation = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum            = true
  }
}

And finally, it goes through 🎉



❯ terraform init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

Terraform has been successfully initialized!

So we can store states, that’s great, right? The configuration is a bit verbose, and you’ll need to repeat these as well so if you use CDKTF, you can quickly write it as a function:



import { Construct } from "constructs";
import { S3Backend } from "cdktf";

/**
 * Configure a S3 backend for Cloudflare R2
 * You will need to set the following environment variables:
 * - AWS_ACCESS_KEY_ID > xxx
 * - AWS_SECRET_ACCESS_KEY > xxx
 * - AWS_ENDPOINT_URL_S3 > https://xxx.r2.cloudflarestorage.com
 *
 * @param scope This needs to be a Stack
 * @param stateName The name of the state file
 * @param bucketName The name of the R2 bucket
 * @returns
 */
export const cloudflareR2Backend = (
  scope: Construct,
  stateName: string,
  bucketName: string
): S3Backend =>
  new S3Backend(scope, {
    bucket: bucketName,
    key: stateName,
    region: "us-east-1",
    skipCredentialsValidation: true,
    skipMetadataApiCheck: true,
    skipRegionValidation: true,
    skipRequestingAccountId: true,
    skipS3Checksum: true,
  });

⚠️ At the time of writing, all the S3 backend configuration keys were not backported to CDKTF. They’ve been updated with this PR and will land in the v0.20 release. You can install a pre-release in the meantime.

Alright, this was fun right, we’re done, right? Well… Not exactly. You see, one key feature of the AWS S3 Backend is that it also supports state locking via DynamoDB, a NoSQL datastore. These locks prevent others from overwriting the state file while you’re also doing something on the stack and are pretty much needed when multiple people work on the same projects. If you need this feature, please read on.

The Worker way

Cloudflare does not have a DynamoDB API compatible service, so if we want to have lock support, we have to either implement some kind of Frankenstein DynamoDB API Worker for Terraform to use it (which sounds like a fun weekend to be honest), or evaluate another backend. In this case, we’ll look at the HTTP backend and implement it with a Cloudflare Worker, backed by R2.

Design

The HTTP backend needs an API service to implement the basic GET, POST, and DELETE methods. For the lock feature, it can use the LOCK and UNLOCK methods from RFC2518 (WebDav). Not all frameworks and HTTP servers might support these unusual methods so they are configurable by Terraform. In our case, no need to do that, they will work natively.

So we will develop a quick Worker to manage Terraform states with R2 storage. What about our locks?

You might be aware that Cloudflare has a KV value product which could be a tempting use case for the locks and it was my first thought but in the case of locks, consistency is important (to avoid race conditions if you have a coworker in the US and another in Asia), and only R2 has a strong consistency model so we will also use it to store locks. (I’m also aware of the Durable Objects product, it could be an interesting alternative!)

Writing a worker

I’ve been wanting to write a Worker for quite some time and already had a library on my radar: Hono

It is a full-featured HTTP router with support for Cloudflare Workers and so onward I went with it and created my project like so: npm create hono@latest tf-state-worker

This will set up a basic project and configure wrangler, a Cloudflare tool to develop and deploy your brand-new worker.

From then, it was a matter of implementing the API (which is not exactly documented, a lot of reading Terraform source code) for reading and writing states, as well as the locking logic:



export const statesRouter = new Hono<{ Bindings: { STATE_BUCKET: R2Bucket }>();

statesRouter.get('/:stateId{[a-zA-Z0-9][\\w\\-\\.]*}', async (c) => { ... });
statesRouter.post('/:stateId{[a-zA-Z0-9][\\w\\-\\.]*}', async (c) => { ... });
statesRouter.delete('/:stateId{[a-zA-Z0-9][\\w\\-\\.]*}', async (c) => { ... });
statesRouter.on('LOCK', '/:stateId{[a-zA-Z0-9][\\w\\-\\.]*}', async (c) => { ... });
statesRouter.on('UNLOCK', '/:stateId{[a-zA-Z0-9][\\w\\-\\.]*}', async (c) => { ... });

As a bonus, I quickly wrote a listing endpoint, which could also probably send some HTML later:



❯ curl https://tf-state-worker.xxx.workers.dev/states
{"states":[{"id":"states/foo.tfstate","size":247,"uploaded":"2024-01-07T22:21:52.030Z"}],"locks":[]}

The result

The resulting worker can be found at https://github.com/adrien-f/tf-state-worker and you can follow along the README to deploy it on your Cloudflare account. As an interesting exercise to you dear reader, I suppose with a few tweaks this could be deployed as a Deno worker with their KV storage solution.

Clone the repository on your development environment, install everything then move the wrangler_example.toml to wrangler.toml, and finally run npx wrangler deploy to get an URL pointing to your new worker:



❯ npx wrangler deploy
 ⛅️ wrangler 3.22.3
-------------------
Your worker has access to the following bindings:
- R2 Buckets:
  - STATE_BUCKET: tf-states
- Vars:
  - AUTH_PLUGIN: "fail"
Total Upload: 55.37 KiB / gzip: 12.93 KiB
Uploaded tf-state-worker (3.06 sec)
Published tf-state-worker (6.00 sec)
  https://tf-state-worker.xxx.workers.dev
Current Deployment ID: xxx

Aaaand wait a minute, no security? Dear astute reader, we're indeed missing something quite important. We do not want our worker to accept any requests to read and write anything in your R2 bucket, especially since states could contain sensitive data 😱!

Since security could be implemented in many ways, by default the worker will reject all requests. There is a plugin system with the first one being a basic username/password combo, but JWT support is planned. By default, the worker will not work until an implementation is configured.

As an alternative, you could also put that worker behind Zero Trust or an mTLS-secured route with API Shield. The choice is yours.

With that done, it’s just a matter of configuring your backend:



terraform {
  backend "http" {
    address = "https://foo:bar@tf-state-worker.xxx.workers.dev/states/foo"
    lock_address = "https://foo:bar@tf-state-worker.xxx.workers.dev/states/foo"
    unlock_address = "https://foo:bar@tf-state-worker.xxx.workers.dev/states/foo"
  }
}

And there we go, a functional Terraform state solution with locks:



❯ terraform apply -auto-approve

Acquiring state lock. This may take a few moments...

Changes to Outputs:

  + foo = "bar"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

foo = "bar"

What’s next?

On the TODO list I have for the worker is a better HTML view, some audit logs, webhook support, and more. Contributions are super much welcomed so let’s get in touch in Github's issues 🙂

Terraform also has a Remote backend which is used by some external vendors like Jfrog to store state, but implementation documentation can not be found and I've reached my limit of reverse-engineering APIs for the weekend, until next time!

Rollbacking Helm releases from an UI with Windmill - Part 2

Adrien F — Sun, 10 Dec 2023 12:48:59 +0000

Welcome back to part two of our Windmill series! Today, we'll learn how to chain together our scripts into a flow and trigger it using the user interface we made before. Let's begin!

Creating a test release

To avoid disrupting too much our Kubernetes infrastructure, we’ll install a test release so we can play with it:

helm upgrade -i test-a oci://ghcr.io/stefanprodan/charts/podinfo
# another time so we can rollback it
helm upgrade -i test-a oci://ghcr.io/stefanprodan/charts/podinfo

And refreshing our UI, the release will appear ready to be controlled from our table.

Going with the flow

To showcase the last (and not the least) feature of Windmill, let’s put together a script to rollback a release and then send a notification on Slack, we’re going to build a Flow!

Let’s start by clicking on the “+ Flow” button on our homepage, and we’ll start by giving it a good name, and then click on the “Input” box on the diagram on the left.

We’ll be prompted for the inputs of our flow. You can go a few ways, one really nice feature is that Windmill can open an endpoint to which you can POST data, which will then get converted automatically into inputs. But in this case, we’ll need only two inputs: the release name and its namespace.

And it will look like this:

With that, let’s click on the “+” button just below inputs to add our script:

We’ll create a new Bash inline script:

# shellcheck shell=bash
# arguments of the form X="$I" are parsed as parameters X of type string
helm_release="$1"
namespace="$2"

out=$(helm rollback -n "$namespace" "$helm_release")
status=$?

jq --null-input --arg output "$out" --arg status "$status" \
    '{"output": $output, "status": $status}' > ./result.json

In this script, we take our two inputs, store the rollback command and its status in variables then use JQ to quickly construct a JSON valid output. This will enable us to handle any errors.

And, as for the UI, by clicking on the “Plug” connect icon, we can pass our flow inputs to this step:

Before testing the rollback, we will need to add new permissions to our workers so they can actually do the rollback. This step will be left to your discretion.

⚠️ This is all for the sake of the demo but remember to properly separate the workers into their own pool with their proper permissions and lock down on their usage.

Now that we have the rollback, let’s also send a notification to Slack! You will need to create a new application for your workspace, the Quick Start guide is the recommended way. Come back when you have an App Token (it should start with xoxb)

Let’s click again on the “+” sign after our rollback script, and let’s do a quick search on the community hub for a script to send a message, you can directly search for “Send message to channel (slack)”. If you’re prompted to add inputs to your flow, reject it.

This community script is the perfect occasion to showcase another feature of Windmill: “Resources”.

A resource is simply a connector to another service like Datadog, AWS, SQL database, etc… It’s a great way to share scoped accesses to services inside a script. Here, we’ll go ahead and click on the “+” sign next to the Slack Resource input, paste our App token and save it.

We’ll also provide the channel and write a templated text message, and since it’s a JS expression we can interpolate some variables:

`Rollbacked \`${flow_input.helm_release}\` in namespace \`${flow_input.namespace}\``

It should now look like this:

However, if our rollback fails, we may want to handle this error and send a different Slack notification. We could do this using a ternary condition in our 'text' input. But for the sake of this example, let's create a branch. Click on the "+" sign before our Slack script and select the "Branch to one" option. To handle cases when our status is not 0, we'll edit the predicate. We'll then duplicate and align our Slack messages accordingly:

Note: in practice you would fork the Slack script and handle the status directly with JS for example, this way you could also customize the message sent using Slack Blocks.

Let's test our new flow. If everything was configured correctly, we should see the result window:

Finally, let's integrate this into our UI to wrap up for the day! Deploy the flow and then return to the App UI editor. Here, select our table component. On the right, there's a section titled "Table Actions", where we can set up various interactions with our table rows. For now, let's add a new button.

You can now rename the button, change its color, and even add an icon. Once you're satisfied, select "Event Handler → Select a script or flow" to choose the Helm Release flow we created earlier. Then, you can input the release and namespace based on the table row. The final result should look like this:

At the end of the settings, make sure to toggle the recompute of the table so changes are reloaded! And now, if all goes well, you should have a shiny UI with a functional rollback buttons, and Slack notifications 🎉

When you click on Rollback, the following notification will be sent to your Slack channel:

Wrapping up

That's all for now! If you enjoyed this series, let me know and we can certainly plan another deep dive into the flow feature. We can explore how to connect more data sources to build more complex user interfaces! I hope you enjoyed this presentation of Windmill. See you soon!

Building a no-code Helm UI with Windmill - Part 1

Adrien F — Wed, 06 Dec 2023 15:28:08 +0000

Hey everyone! Today I would like to introduce you to Windmill, a serverless platform enabling you to write scripts in many languages, chain them together, build an UI around them and much, much more. Sounds too good to be true? Well let’s put it to the test by building together a dynamic UI for managing Helm charts in a Kubernetes cluster.

Introduction

There's no doubt that you've found yourself in a situation like this at some point: you’ve been asked to build a quick workflow, where you had to chain a few actions, send a notification somewhere or a piece of data into a system. You have the code ready but then, all of that needs to be packaged and deployed (after tests), and if you’re lucky, you already have all the resources and systems for that part of the process.

But releasing your code into the world is only the initial step in a larger process. Once the code is out there, it needs to be scheduled, executed and observed. This doesn't just happen on its own and you need systems in place to do just that.

This is where serverless Cloud solutions such as “AWS Step Functions” caters to these needs. Step Functions allows you to chain Lambdas, and call upon AWS services with extra logic to your workflow. In essence, it provides a robust platform that allows your code to function as intended, while also offering the necessary tools to monitor and optimize its performance.

See https://aws.amazon.com/fr/blogs/aws/new-aws-step-functions-workflow-studio-a-low-code-visual-tool-for-building-state-machines/ for more info!

Cloud solutions are a great fit indeed, and they’ll probably be good enough for many use cases. But today we’ll talk about an entire platform centered around these workflows, a platform where you could share actions with your coworkers, tie them together in workflows and build UIs around them.

And that’s where Windmill comes in, with three key concepts:

Scripts, which are re-usable pieces of code that have inputs and outputs
Flows, which tie scripts together with logic, branching and retries
Apps, made of dynamic React components which can trigger and display scripts and flows

With these three building blocks, you’ll be able to create powerful and reliable workflows, let’s put it to the test.

I’ve created a local cluster with K3S and installing Windmill could not be simpler with just one chart to configure, which already has sane defaults to get started. For this demo we will also configure workers to passthrough environment variables to our scripts so that they have access to the Kubernetes API server for later.



helm repo add windmill https://windmill-labs.github.io/windmill-helm-charts/
helm show values windmill/windmill > values.yaml
yq -i '.windmill.workerGroups[].extraEnv += {"name": "WHITELIST_ENVS", "value": "KUBERNETES_SERVICE_HOST,KUBERNETES_SERVICE_PORT"}' values.yaml
helm install windmill windmill/windmill --namespace=windmill --create-namespace -f values.yaml --set windmill.appReplicas=1 --set windmill.lsbReplicas=1

Note: see the Chart’s README for all potential values. I recommend managing them finely, with dedicated service accounts for example.

We’ll then watch the pods being created:



$ kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
windmill-app-567858d877-gqp26              0/1     Running   0          32s
windmill-lsp-7bd57bfdfc-bdfg8              1/1     Running   0          32s
windmill-postgresql-0                      1/1     Running   0          32s
windmill-workers-default-dc4cc4d76-5wvtl   1/1     Running   0          32s
windmill-workers-native-8df7667f-6xl29     1/1     Running   0          32s                   0/1     ContainerCreating   0          20s

While we wait, let’s explore a bit more the architecture of the project and what’s being deployed:

We have a PostgreSQL instance starting
An app deployment for the UI and API
Several workers deployments to execute our scripts
And an LSP server to provide autocompletions

Once running, you’ll notice an Ingress configuration has also been created, you can adjust the hostname either via values or via the initial configuration steps. Open a browser to your ingress and you’ll be greeted with the default login screen, it will be admin@windmill.dev:changeme:

And after a few settings that you can tweak, you’ll be prompted to change the admin & password email and finally you’ll be able to create a new workspace that will contains all your workflows.

⚠️ There is an opt-out Telemetry option you might not want to miss if testing in a sensitive environment, but it’s also helpful data for the project to grow!

And there you go, we’re in !

Building an Helm UI

To showcase the basic set of features of Windmill, let’s build ourselves a quick Helm UI, that our users could then use to list and rollback releases. On rollback, we’ll also send a notification to Slack.

Listing namespaces

We’ll need a script to list our cluster namespaces, and then the releases it contains.

Click on the “+ Script” button at the top of the window, we’ll select Python as the runtime, give it a good name, and you can click on the editor on the left to get the Settings window out of the way.

We’ll import the kr8s package to interact with the cluster, and when you’ll remove arguments from the main function, you will also see them disappear from the input section! Quite neat.

Let’s write a script to return a list of current namespaces:



import kr8s

def main():
    return [ns.name for ns in kr8s.get("namespaces")]

And by pressing CTRL+Enter, execution will start and will error-out:

Which should not be surprising as we did not configure additional permissions to Windmill workers.

Let’s add a quick ClusterRole and ClusterRoleBinding to the windmill user. In practice, we would start adding a custom ServiceAccount for this dedicated worker pool:



apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: windmill-k8s
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: windmill-k8s
subjects:
- kind: ServiceAccount
  name: windmill
  namespace: windmill
roleRef:
  kind: ClusterRole
  name: windmill-k8s
  apiGroup: rbac.authorization.k8s.io

Re-running the job, we will see the expected result:

Note the "found cached resolution" debug line. It indicates that Windmill computed and cached all the dependencies necessary to run this script. This will prevent subsequent pip install actions until changes are made.

You can now click on the Deploy button at the top right and CTRL+ENTER on your keyboard to execute the script, where you'll get more details on the duration, memory usage and status.

Listing Helm releases

Since Helm is included by default in the worker's Docker image (thanks for that!), we're able to create a new Bash script:



# shellcheck shell=bash
namespace="$1"
helm list -n $namespace -o json > ./result.json

Our script is accepting an argument “namespace” and we’ll write the result to result.json which will be parsed by Windmill and made available.

We will now have to adjust our RBAC permissions to grant read on the secrets:



kubectl patch clusterrole windmill-k8s --type='json' -p='[{"op": "add", "path": "/rules/-", "value": {"apiGroups": [""], "resources": ["secrets"], "verbs": ["get", "list"]}}]'

And by saving and running our script, we’ll see our list of releases:

Notice how we automatically had an input field for our namespace? Windmill automatically built a quick UI for our script so we're able to run them whenever we want.

Building an Helm UI

With our first scripts ready, we are now able to chain them together to build an UI! Back on the homepage of the workspace, let’s click on the “+ App” button which will open the App editor.

There are many things to present of this editor, but I will refer you to the Youtube playlist made by Windmill authors and the documentation website.

For now, let’s focus on our quick & dirty! Scroll down the list of available components and drag and drop the “Select” component.

Now we need to prepare the list of namespaces for this component. At the bottom of the editor you will find the “Background Runnable” section, click on the + (plus) sign to create a new runnable, click on “Select a script or flow” then on the opened window, select the “Workspace scripts”, then finally, your script listing namespaces.

Note: you might have not missed the “Frontend” language section when creating a new background runnable, you can indeed add extra Javascript code that will be executed in the browser.

The Select input form expect data in a specific format, so we’ll add a transformer, which is just a quick Javascript function to transform our namespaces into what’s expected. When selecting the runnable, you will notice on the right in the settings panel, a “Transformer” section.

Click on the “Add” button and in the new script view, transform the result then click on “Run”:



return result.map((result) => ({label: result, value: result}))

You can inspect the current state of the app and view our reflected data model.

Let’s now plug our Select input. Click on the component, and on its configuration tab, click on the little plug icon next to “Items”, this will allow us to configure the data source for this select. Click on the “result” button in the bg_0 output and the select will be automatically populated!

Next up, we’ll do the same thing with the Helm releases. We create a new background runnable, we select our “List Helm releases” script and this time we’ll notice we’re expected to configure the input. Again, click on the “Connect” plug symbol and we’ll select the result of our select component in the Outputs panel at the left (or you can also write a.result) if you’re in a hurry. Make sure to disable the “Run on start and app refresh” option as we’ll need to have the namespaces first.

Now, every changes to the select value will make Windmill execute again the script with the proper variable, making that data available. Let’s display it with the Table component!

Again, drag & drop the Table component, and select the plug symbol. Select the result of bg_1 which is our list of Helm releases, and boom, you’ve got a table with your releases:

When you select another namespace, the table will refresh with the releases from this namespace. Adjust width and re-organization or add more components if you wish, then you can click on Deploy to get a pretty view that you can share with coworkers:

Great success! You can re-order columns or even hide some if you want by configuring the component.

Wrapping up

And that's it for this first part! I hope I was able to share with you the same excitation I had when I first started using this tool! In the next and final part, we'll look at how we can implement a Flow and add a Rollback button to our table. See you!

Resources

The following resources might be of interest:

Windmill Youtube channel
Windmill documentation
Trevor Sullivan has made a series of videos using Windmill if you want to deep dive