DEV Community: Chris Heppell

Next.js development with instant dev-database provisioning

Chris Heppell — Mon, 05 Oct 2020 09:39:54 +0000

Next.js is a really exciting framework that makes it simple to build React applications with excellent features that you’d expect in production. I won’t go into much detail here, as the Next.js documentation does that brilliantly, but one of the key things that stood out is the easy development experience.

By using create-next-app you can immediately get set up with an interactive dev experience that auto-reloads code changes in your browser as you make changes. Next.js also has support for building web APIs without needing any separate API server (e.g, ASP.NET Core, Go’s net/http package). It can serve your frontend, and handle your API requests all in one.

But what if your backend API needs to request data from a database? Well, this is easy enough to do via packages like serverless-mysql that let you initiate connections to a MySQL database from NodeJS. Add the dependency, run your queries in response to an API call in Next.js and then send the results back to the frontend (usually as JSON). Easy.

Something is missing…

Next.js itself is great. From a dev perspective it’s easy to get set up instantly with an environment and start coding.

But when it comes to building an app that requires a backend database, things get a bit more tricky as a developer.

In production, you’ve likely got some kind of managed database offering (or a DBA) that is responsible for maintaining your database. You get given some connection details and off you go.

But as a dev who wants to get started actually making changes, you might start to ask yourselves these kinds of questions:

Should I install the database engine on my machine?
Should I run it in a Docker container?
If I run it in Docker, how do I persist my data in the case of container restarts?
Should I connect to a database instance I share with the rest of the development team?
What if I need production-like data? Restoring a backup can take a long time…

If you’ve set up Next.js but you’ve hit this roadblock, you might be frustrated with how good the development experience was up to this point. Suddenly, you have to worry about maintaining a stateful database alongside your dev environment, and that’s not insignificant.

Levelling the playing field

Getting a database environment should be as easy as scaffolding an entire Next.js application. So we set about making that reality.

Spawn is a new solution we’re working on that aims to make it really easy to include databases in your dev and CI environments. It removes the uncertainty around how to get a database environment and instead encourages you to spin them up and down as much as you like, without any concern over infrastructure.

To really put this into perspective, let’s see how Spawn fits in a Next.js application.

If you’d like to check out the code I’ll be discussing, you can review the GitHub repository here.

Hooking things up

For the sake of brevity, I followed the tutorial on setting up a Next.js app with a MySQL Database That Builds and Deploys with Vercel. This is an excellent tutorial which I encourage you to check out, but we won’t be concerning ourselves with the deployment side for now.

The real question is how do we spin up that database and hook it up with the API that Next.js makes available to us?

Well, Spawn is entirely commandline driven. As such, it’s possible to configure an npm script that runs just before the dev script provided by next.js. This predev script does one thing: create a Spawn data container and update the .env.local file with the connection details to be consumed by our API.

Here’s a snippet of the spawn.sh script:

if containersExist; then
    echo “Spawn data container already provisioned”
else
    echo “Provisioning Spawn data container for dev”
    spawnctl create data-container --image $DATA_IMAGE \
      --name $CONTAINER_NAME -q
    echo “Spawn data container successfully provisioned”
fi

echo “Updating data container connection details in .env.local”

db_conn_json=$(spawnctl get data-container $CONTAINER_NAME -o json)

DB_HOST=$(echo $db_conn_json | jq .host)
DB_PORT=$(echo $db_conn_json | jq .port)
DB_USERNAME=$(echo $db_conn_json | jq .user)
DB_PASSWORD=$(echo $db_conn_json | jq .password)

cat << EOF > .env.local
MYSQL_HOST=$DB_HOST
MYSQL_PORT=$DB_PORT
MYSQL_USERNAME=$DB_USERNAME
MYSQL_PASSWORD=$DB_PASSWORD
MYSQL_DATABASE=nextjs
EOF

As you can see at the end of this, we write the database connection details to the .env.local file. This presumes that I already have a data image available (which can be created from SQL Scripts, or the latest backup from production if you care about production-like data) and then creates a data container I can use which is a copy of that data image. This data container is created instantly — regardless of the size of the data image.

There’s absolutely nothing else for me to do here. Instead of needing to install Docker or a database engine on my machine and start it up, wait for connections to be accepted, then figure out the correct connection details to pass through, all I have to do is ask Spawn to give me a dev database and a few seconds later I’ve got the connection details.

This makes it much easier to bootstrap an entire dev environment including the database.

The best bit? All of this is committed to source control. After installing the spawnctl, I can checkout the repository on my machine, run yarn dev and then I’ve got my Next.js application up and running, backed by an API which can communicate with my database instantly. Onboarding a new dev is as simple as asking them to clone your repo.

Spawn has a bunch of other benefits, like being able to save the state of my database as I work in case I accidentally mess up and delete some data, and I can even make as many save points as I like to create a history of revisions. Equally, each dev in my team gets their own isolated data container for use in development, so I don’t have to worry about sharing a single instance and overwriting others’ changes.

If you’d like to take a more detailed look at the code to set this up, you can view it in the GitHub repo.

Get Spawn now

We’re currently accepting beta users for Spawn, so if this workflow looks like something you’d be interested in you can sign up to get access here.

Extending Kubernetes with Operators

Chris Heppell — Wed, 01 Apr 2020 14:48:30 +0000

This article was originally posted to the Ingeniously Simple Medium publication.

Having attended the last two European KubeCon events (2018 and 2019), it’s become increasingly obvious that operators are becoming a hot topic within the community.

There were zero sessions on operators in 2018, whereas there were 9 in 2019. There’s even a dedicated OperatorCon hosted by Loodse at this year’s (unfortunately postponed) KubeCon!

I also had the pleasure of attending the Operator Framework Workshop session delivered by Red Hat. This was an excellent session which covered the basics of Operators and how to create them using the Operator Framework.

So what is an operator?

The CoreOS documentation does an excellent job of explaining this, so I’ll only try and summarise it here.

An operator is an application that runs in Kubernetes, acting upon the presence of Kubernetes Custom Resources based on their definition. Typically, you might use an operator to handle the management of services within Kubernetes by encapsulating the operational knowledge of a given service in code. Ultimately, the operator runs as a pod within the cluster (possibly with more supporting pods).

As an example, take a look at the Zalando Postgres Operator. Running databases in Kubernetes can be tricky, since it requires persistent volumes, stateful sets, and services to expose them. Instead, the Zalando Postgres Operator encapsulates all the complexity of this behind a single “custom resource” in the cluster of type “postgresql”. You can see an example of a minimal postgresql configuration custom resource here.

Notice how it defines things such as the users and databases you want in the instance when it starts. Creating the users and databases is possible without an operator provided that the container image you use supports it. If it doesn’t, you’ll need to explore different options for initialising the database after the pod enters a ready state. Furthermore, you’ll have to handle the persistent volumes, statefulsets, and services yourself, including how all of those link up with each other.

Instead, the operator will create all of these resources for you without you needing to know the details. All you do as a consumer of the operator is create the postgresql custom resource and submit it to the cluster. The operator will do the rest — creating the associated resources in the cluster for you and performing any post-setup actions you defined. It can also (optionally) do things you already come to expect from Kubernetes such as deployments ensuring the correct number of pod replicas are always available. The details depend on the operator implementation.

In summary, the operator is effectively giving you a higher level abstraction over Kubernetes from the point of view of running and managing Postgres.

Operators add a level on top of the existing Kubernetes API — letting you work at a higher level of abstraction

How do you build an operator?

Whilst operators are a recognised pattern for building these solutions, there are also some opinionated frameworks that make it easy to get started with creating one.

The one I’ve had experience with is the operator-sdk, which is part of the Operator Framework.

This scaffolds all the boilerplate required for creating a Kubernetes Operator. It autogenerates things such as the custom resource definitions, the controller for consuming those custom resources, and all of the plumbing to communicate with the Kubernetes cluster itself.

The operator-sdk documentation is the best place to understand how to actually generate all these things, so I won’t repeat it here. If you’re looking to build an operator yourself, this is an excellent place to start.

How have we used operators?

One of the projects we’re working on at Redgate is Spawn. The idea of this is to provision databases for dev/test purposes instantly, irrespective of the underlying size of that database. You can also save arbitrary states of the database, instantly reverting destructive changes or jumping between known checkpoints to support and accelerate development.

We use a combination of different technologies within Spawn, but one of the key components is the Spawn Kubernetes Operator.

“spawncontainer” Kubernetes Custom Resource Definition (CRD)

This is responsible for provisioning database pods in a Kubernetes cluster according to a spawncontainer custom resource submitted to the cluster. The spawncontainer custom resource defines a few things in its specification:

The database engine to provision
Database engine-specific configuration parameters (e.g, admin credentials)
The network location of the filesystem to use as the underlying filesystem for the database

This provides a nice abstraction for provisioning those databases. Typically, you’d have to handle creating a pod, with a mounted persistent volume claim that refers to a persistent volume, and then exposing that pod with a service.

Instead, the spawncontainer custom resource is submitted to the cluster with the desired configuration and the Spawn Operator consumes that, constructing and connecting all the necessary underlying Kubernetes primitive resource types. When it comes time to clean up this resource, we don’t have to worry about deleting each individual Kubernetes primitive. We simply delete the spawncontainer custom resource, which causes a cascade of deletions for all primitive resources that were created as a result.

This dramatically simplifies things for the supporting services that compose Spawn. They simply request a spawncontainer, and then at some point a database engine is available configured appropriately. The complexity of Kubernetes is hidden behind this abstraction, and operations are performed on the custom resource directly.

Building this operator has made developing Spawn significantly easier. We can take advantage of many of the benefits of Kubernetes without having to undertake constructing hand-crafted requests to the cluster API and figuring out how to watch resources and react to changes. Instead, we rely on the scaffolding from the operator-sdk and simply concern ourselves with writing the business logic — the piece that constructs and links all the Kubernetes resources together that represent a spawncontainer custom resource.

Conclusion

Operators are an excellent way to build applications that extend the Kubernetes API with domain-specific knowledge. You can hide the complexities of running a system by putting it behind a custom resource that represents the desired state of the system. Users simply interact with that object, and the important changes are made by the operator which effectively translates the domain specific resource into Kubernetes primitive resources.

If you’re looking to do this, then I’d recommend checking out the operator-sdk and having a play to see what you can do. If you’re trying to run a service in the cluster, then operatorhub.io is a great place to go to see if an operator already exists for managing it.

Thanks to our operator, we’ve also just launched a closed beta programme for Spawn. If you’re wanting to accelerate your dev/test workflows with instant isolated copies of large production-like database environments that you can immediately spin up, tear down, checkpoint and reset, then sign up at spawn.cc/beta.

You can also read more about Spawn here.