DEV Community: jxtro

Kubernetes vs Philippine Power Outages - On setting up k0s over Tailscale

jxtro — Mon, 01 Jul 2024 16:35:38 +0000

Building a reliable IT system in the Philippines presents unique challenges such as frequent power outages and unreliable internet connectivity.

To address these issues effectively, our team has implemented a resilient setup ensuring uninterrupted access to critical services for our end-users.

This guide will walk you through a similar setup using Tailscale and k0s, which can be replicated in your homelab environment. If you're curious only about the setup, feel free to jump to Section II.

I. The Challenge

To give you some background, our team manages multiple projects for various clients, hosting most services on local servers near them. However, a significant issue we face is frequent power interruptions due to maintenance or emergencies at local substations. These disruptions occur nearly every week, sometimes lasting 8-12 hours, effectively rendering our services unavailable for entire workdays.

Side-note: Why not host on cloud?

Well, it's a bit complicated. Some services work well in the cloud (so we've put them there), but others have their own unique needs. For example, let's take a closer look to two of our main projects: Impactville and Lupain.AI.

Impactville deals with private data from organizations that prefer not to store it in the cloud due to privacy concerns. Meanwhile, Lupain.AI handles sensitive land data from local government units, requiring secure and local storage.

From a cost perspective, Lupain.AI involves intensive processing of satellite data. Using cloud resources could drive up costs, especially with the increasing USD-PHP exchange rates. It's more cost-effective for us to manage these tasks using a self-hosted cluster of GPU nodes.

Back to the challenge

To summarize our scenario:

We have full control and ownership of nodes distributed across the country.
Our goal is to achieve fault-tolerant services with minimal downtimes (ideally less than 1 minute, with 5-10 minutes being acceptable).
Data redundancy and high availability are essential requirements.

Given these needs, the most viable solution is to set up an orchestrator capable of detecting downtimes, automatically rescheduling services, and distributing them across a cluster of nodes.

In this setup, if one server experiences a power outage, the services will be temporarily shifted to other servers until normal operations resume.

Therefore, adopting Kubernetes is an obvious choice for us. Kubernetes is an open-source system designed specifically for automating deployment, scaling, and management of containerized applications.
This guide will walk you through the basic setup of deploying your own Kubernetes cluster using k0s and Tailscale.

Disclaimer: The setup described here is simplified and may differ from our production setup. Our production environment addresses various complexities such as ISP DNS issues [1][2], rate-limiting, weather-related challenges for Starlink-connected nodes, server security hardening, encryption of redundant data, and cluster ingress. These topics require detailed discussions and are either reserved for future posts or treated as internal know-how.

For a small homelab setup, however, this guide should provide sufficient guidance.

Kubernetes setup

In this guide, we'll set up a Kubernetes cluster using k0s and connect our nodes via Tailscale. Here's an overview of the technologies involved:

k0s k0s is an open-source Kubernetes distribution designed for simplicity and versatility. It includes all necessary features to build a Kubernetes cluster and is lightweight enough to run on various environments such as cloud, bare metal, edge, and IoT devices. Its minimal setup and easy configuration make it ideal for our needs.

Tailscale While any VPN could be used, Tailscale stands out for its ease of setup, comprehensive documentation, and reliable networking capabilities. MagicDNS, which simplifies DNS management, adds an extra layer of convenience.

Distributed storage For distributed storage, you have various options to choose from. The simplest approach is setting up NFS or NAS, though configuring it for high availability (HA) may be required. In our setup, we've chosen to use SeaweedFS, a distributed storage system that provides scalability and efficient management of large data volumes. Note that configuring SeaweedFS for HA is beyond the scope of this guide.

Instructions

To begin setting up your Kubernetes cluster, follow these steps:

1. Manual node setup

First, ensure SSH is configured securely to access your nodes. Verify you have SSH access to all nodes and that they use key-based authentication. Disable password authentication temporarily for cluster setup; you can re-enable it later in your SSH configuration.

2. Connect them over VPN

Next, establish a secure connection between your nodes using Tailscale:

Sign up for Tailscale and add your devices to the network. Check https://tailscale.com/
Follow Tailscale's instructions to install and connect Tailscale to your network. See https://tailscale.com/kb/1017/install
Check that tailscale0 appears in your network interfaces (e.g., via ifconfig).
Ensure your control machine, where you'll run kubectl, is also connected to the Tailscale network.

3. Install k0s

To set up your Kubernetes cluster, follow these steps:

Install k0s in your control machine. Begin by installing k0s on your control machine. You can find detailed instructions at k0s Installation. Use the command:

curl -sSLf https://get.k0s.sh | sudo sh

Use k0sctl for automated deployment: To streamline the installation across nodes, use k0sctl:

Install k0sctl depending on your OS. Refer to k0sctl Installation:

brew install k0sproject/tap/k0sctl
choco install k0sctl

Generate a k0sctl configuration file: Create a k0sctl configuration file to define your cluster setup:

k0sctl init > k0sctl.yaml

Customize the configuration as needed. Here's an example configuration:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - role: controller
    ssh:
      address: 10.0.0.1 # replace with the controller's IP address
      user: root
      keyPath: ~/.ssh/id_rsa
  - role: worker
    ssh:
      address: 10.0.0.2 # replace with the worker's IP address
      user: root
      keyPath: ~/.ssh/id_rsa

For further customization, refer to the k0sctl Configuration Documentation.

Bootstrap the Cluster: To initialize and deploy your Kubernetes cluster, execute the following command:

k0sctl apply --config k0sctl.yaml

k0sctl will automatically install and deploy k0s on the designated machines within your network, configuring the Kubernetes cluster for operation. Once deployed, generate the kubeconfig file to manage the cluster using kubectl:

k0sctl kubeconfig > kubeconfig
kubectl get pods --kubeconfig kubeconfig -A

Uninstall or Reset the Cluster: If you need to reconfigure or remove the cluster, use the following command:

k0sctl reset --config k0sctl.yaml

This command resets the cluster configuration, allowing for subsequent deployments or modifications as needed.

4. Configuration

While the previous instructions provide a standard setup, additional configurations are necessary to integrate with Tailscale and manage private container registries effectively.

Tailscale-connected Nodes

To ensure proper IP assignment by k0sctl for Tailscale-connected machines, specify the correct network interface in the configuration. For Tailscale, use tailscale0. Here’s an example configuration snippet:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
  hosts:
    - ssh:
        address: machine-1
      privateInterface: tailscale0
      role: controller

    - ssh:
        address: node-2
      privateInterface: tailscale0
      role: worker

Private Container Registry

If your application's images needs extra privacy, chances are you're storing container images in a private registry. To ensure that k0s (specifically, containerd) pull the image from the right registry, follow the these instructions:

Create a custom configuration file for containerd on each worker node:

sudo nano /etc/k0s/containerd.d registry.toml

Add the following content:

[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

This directs containerd to look for hosts in /etc/containerd/certs.d.

Create a hosts.toml file for your registry domain or IP

sudo nano /etc/containerd/certs.d/<registry-domain or ip>:<registry port>/hosts.toml

Populate it with:

server = "http://<domain or ip>:<port>"

[host."http://<domain or ip>:<port>"]
  skip_verify = true # If your registry is not configured via TLS

Note: For production environments, ensure TLS certificates are correctly configured. Refer to containerd documentation for additional configuration details.
Once configured, k0s will utilize these settings to pull private images from your registry as needed.

Networking

Network configuration has proven to be quite challenging, consuming considerable time and effort as we troubleshooted various issues with our server setups. After days of painstaking work, here's what we've uncovered.

For networking, k0s supports various providers for managing inter-pod networking, known formally as a Container Network Interface (CNI). For more detailed information about k0s networking capabilities, you can refer to the official documentation here.

By default, k0s uses the kube-router CNI, known for its lightweight and performant nature. However, we encountered specific issues where inter-pod communication between nodes failed. Key diagnostics such as nslookup failing to connect to nameservers and traceroute showing asterisks led us to investigate further.

After extensive troubleshooting involving iptables, we determined that kube-router was not utilizing the correct interface for communication—in our case, Tailscale. Additionally, kube-router does not currently support explicitly setting the network interface and may not add this functionality in the near future (refer to GitHub issue #567). As a result, we've made the decision to transition to a different CNI.

Another built-in CNI option for k0s is Calico, which offers more flexible configuration options, including network interface settings. If you're encountering issues with kube-router and need to switch to Calico, you can use the following configuration during cluster bootstrap:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  k0s:
    config:
      spec:
        network:
          provider: calico # <-- use Calico
          calico:
            envVars:
              IP_AUTODETECTION_METHOD: "interface=tailscale0" # <-- use tailscale
  hosts:
      - role: controller
        ssh:
          address: 10.0.0.1 # replace with the controller's IP address
          user: root
          keyPath: ~/.ssh/id_rsa
      - role: worker
        ssh:
          address: 10.0.0.2 # replace with the worker's IP address
          user: root
          keyPath: ~/.ssh/id_rsa

It's important to note potential edge cases when integrating Calico with Tailscale as discussed here. To avoid conflicts, we recommend remapping Calico's netfilter packets. This ensures compatibility and smooth operation in your network setup.

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  k0s:
    config:
      spec:
        network:
          provider: calico
          calico:
            envVars:
              FELIX_IPTABLESMARKMASK: "0xff00ff00" # <- use mask
              IP_AUTODETECTION_METHOD: "interface=tailscale0"

After redeployment, pods can now communicate with each other across different nodes!

Node-local load balancing

With the nodes set up, our Kubernetes cluster now handles inter-node communication effectively, even during power outages. However, there's an important scenario we need to address: what happens if the control node experiences an outage? Without a functioning control node, there's no orchestrator to manage pod events, which could lead to downtime for critical services.

To ensure continuous operation, it's essential to plan for high availability of the control plane. This can be achieved by setting up multiple control plane nodes within the cluster.

Fortunately, k0s offers a built-in solution for this with Node-local load balancing. Adjusting a small portion of the configuration allows us to enhance our setup:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  k0s:
    config:
      spec:
        network:
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy

Use your Kubernetes cluster

Now that your Kubernetes cluster is deployed and configured using the steps outlined above, the final step is to set up kubectl, the Kubernetes command-line tool, on your local machine. This tool allows you to manage your cluster effectively.

Follow these steps to complete the setup:

Install kubectl: Install the Kubernetes command-line tool, kubectl, on your local machine. You can download it from the official Kubernetes documentation or use package managers like apt or brew.

Configure kubeconfig: Once your cluster is deployed, set the generated kubeconfig file as your default configuration by copying it to the appropriate directory

cp kubeconfig ~/.kube/config

This step ensures that kubectl uses the correct credentials and configuration to access your Kubernetes cluster.

Verify setup: Confirm that kubectl is correctly configured by checking the status of pods in your cluster

kubectl get pods -A

This command will list all pods across all namespaces (-A flag), indicating that your cluster is operational and ready to deploy applications.

With kubectl configured, you're now equipped to manage and orchestrate containerized applications on your Kubernetes cluster seamlessly!

Final Thoughts

Thank you for reading! I appreciate you taking the time to read up to this point ❤️. If you find any parts confusing or have any issues with replication, I'd be happy to help. Just shoot me an email (thepiesaresquared@gmail.com) or DM/tweet me at @justfizzbuzz.

I thoroughly enjoyed the process of making this guide! If you're interested in more posts like this, I invite you to subscribe to this blog, or let's connect and share our posts on Twitter/X!

Self-hosting Ghost with Docker and PlanetScale

jxtro — Sun, 14 Jan 2024 13:27:07 +0000

PlanetScale and Ghost were previously incompatible due to differences in their support for foreign key constraints. With PlanetScale now supporting foreign key constraints, a seamless collaboration between the two is achievable. Nonetheless, there remain minor incompatibilities that require resolution.

The first part of this post will show you how to set up Ghost using Docker and PlanetScale. In the second part, we'll talk about the issues when putting PlanetScale and Ghost together and how to fix them. If you just want to get things going fast, you can skip the other parts.

ℹ️ This post is originally published on my blog. If you encounter any formatting issues here, you might find better readability on the original post.

Part 1 - Setting up Ghost and PlanetScale

Some warning

For PlanetScale to work with Ghost, make sure foreign key constraints are turned on. It's important to note that foreign key constraints are in beta on PlanetScale. If you decide to revert your database from beta, it could potentially disrupt your Ghost website. Additional information can be found here and here. This is a crucial point to weigh before changing providers.

A. Setting up database

1. Create a database in PlanetScale

Begin by creating a new database in PlanetScale. Follow the official quickstart guide available here. Assign a name to your database and make sure to jot down your connection details, such as the host, username, and password.

2. Activate "Foreign key constraints" in your database.

Ghost requires the use of foreign key constraints to function properly. Head to your database settings by selecting Database > Settings > Beta Features. Then, click on "enroll" next to "Foreign key constraints" to enable this crucial feature.

Great! Let's now move on to setting up Ghost itself.

B. Setting up Ghost

1. Install Ghost.

Installing Ghost can be done in various ways, and you can choose the one that suits you best by referring to its official installation guide. However, for the sake of reproducibility, this guide extends Ghost's official Docker image.

2. Customize your Ghost installation.

At this point, you have the option to add themes and adapters. However, delving into these aspects is beyond the scope of this article.

3. Patch the Ghost installation.

Ghost and PlanetScale have a minor incompatibility, which can be resolved by applying a patch. To do this, create a script named patch.sh with the following content:

https://github.com/JustroX/ghost-docker-planetscale/blob/main/bin/patch.sh

Finally, execute the patch by running:

chmod +x patch.sh
./patch.sh

Please be aware that, at the time of writing, this patch has only been tested with the latest Docker image (ghost:5.75.3) and its effectiveness may vary in future versions (or might become unnecessary). The specifics of the patch will be elaborated upon in the second part of this guide.

For Docker installation, here's the corresponding Dockerfile

FROM ghost:latest
COPY patch.sh patch.sh
RUN chmod +x patch.sh && ./patch.sh

Assuming everything goes smoothly, Ghost should now be configured to work seamlessly with PlanetScale. Now, let's proceed to connect them.

C. Connecting to the database

For configuring the database connection, use the PlanetScale connection details obtained from the previous steps and paste them into your Ghost configuration. Keep in mind that the value for database__connection__ssl should be set to [{"rejectUnauthorized":true}].

database__client=mysql
database__connection__database=<Your Planetscale Database Name>
database__connection__host=<Your Planetscale Database Host>
database__connection__password=<Your Planetscale Database Password>
database__connection__user=<Your Planetscale Database User>
database__connection__ssl=[{"rejectUnauthorized":true}]

The rest of the configuration process is akin to setting up a standard Ghost installation, which you can refer to in the official reference found here.

D. Run your Ghost instance

For Docker installation, launch your Docker container using the following command:

docker build . --tag ghost_example
docker run --env-file ./env ghost_example

Assuming a smooth process, you should observe your Ghost blog initializing and generating the necessary database tables. Once this initialization is complete, your Ghost blog integrated with PlanetScale should be live and ready!

E. Deployment (Optional)

I've created an example repository here for those looking to swiftly deploy a Ghost blog using PlanetScale. Simply follow the aforementioned steps for setting up the database and connecting to it. Beyond that, all that's required is to build and run the Dockerfile.

Github link here

I tested this using Railway, where all that's needed is to provide the repo fork and the necessary environment variables. Railway then automatically identifies the Dockerfile for building and running. The outcome is my blog website.

F. Conclusion

That concludes the setup and deployment guide. Thank you for reading! If any parts are unclear or if you encounter issues with replication, feel free to reach out. I'd be happy to help. Just shoot me an email (thepiesaresquared@gmail.com) or DM/tweet me at @justfizzbuzz.

If you're intrigued by the challenges I encountered while integrating PlanetScale and Ghost, along with the eventual solution in the form of a patch, you can proceed to Part 2. 😄

ℹ️ This post is also published on my blog. If you found this post helpful, you might discover other useful posts there as well!

Part 2 - Integrating Ghost 🤝 PlanetScale

In my view, there are three significant hurdles that both myself and fellow self-hosters faced when attempting to integrate Ghost and PlanetScale:

Configuring the value for database.connection.ssl
Ensuring foreign key constraints are supported by PlanetScale.
Addressing Ghost setup failures during initialization.

A. Configuring the value for `database.connection.ssl`

To overcome this initial challenge, it's crucial to establish a secure connection to the database since it resides outside the same network as the server. Attempting a direct connection without SSL/TLS results in the following error:

While seemingly minor, locating the value for database.connection.ssl can be challenging. Experienced developers might find it obvious, but for those who simply copy-paste from PlanetScale, it may not be as apparent. The community has resolved this configuration value since March last year, as discussed in this forum. From the discussion, the correct value for the configuration should be:

database__connection__ssl=[{"rejectUnauthorized":true}]

B. Foreign key constraints support for PlanetScale.

The second and primary challenge arose from the absence of foreign key constraints support in PlanetScale at that time. This presented a significant hurdle as Ghost heavily relies on this feature. In a Github discussion, PlanetScale's rep said that:

It looks there are at least a couple of big blockers here ... Seems like [Ghost] makes heavy use of foreign key constraints, a feature we don't support on PlanetScale
This is the point where most discussions on integration come to a halt and encounter a roadblock.

Fast forwarding to last month, PlanetScale made a significant announcement regarding the introduction of foreign key constraints support as a beta feature:

Announcing foreign key constraints support — PlanetScale
You can now use foreign key constraints in PlanetScale databases

With this recent development, achieving the integration is now technically feasible, a great time to resume and advance the integration efforts.

C. Addressing Ghost setup failures during initialization

i. The Problem

This particular challenge is a bit tricky. PlanetScale generates a database for you, and all you need to do is establish a connection. However, Ghost encounters startup issues because it erroneously assumes that the database still needs to be created. This manifests as either Ghost attempting to create the database despite its existence, or Ghost failing to detect that the PlanetScale database has already been established.

This concern was previously addressed in this discussion and ended with the following conclusion:

It doesn't appear that there's a ghost CLI option to run the initial setup without attempting to create the database, or any other built-in workaround for the problem you encountered.

To address this, my initial plan was centered around creating a Ghost fork and incorporating an option to skip the database creation step. Luckily, looking back, the solution turned out to be remarkably straightforward, with the bulk of my time dedicated to tracing the error and navigating through the code.

ii. Tracing Ghost's code

My initial strategy involves tracing the source of the error.

Given my limited familiarity with the majority of the codebase, I began following the execution flow of the source code, available at https://github.com/TryGhost/Ghost/blob/main/ghost/core. This journey starts from the initial Ghost initialization and progresses until I stumble upon a piece of code indicating the database creation. I tracked the code references, traversing through files such as index.js -> ghost.js -> boot.js -> and finally DatabaseStateManager.js, until reaching what appeared to be a dead-end.

In DatabaseStateManager.js, Ghost triggers the initialization of the database by invoking the init method from a knexMigrator instance:

  await this.knexMigrator.init();

The knexMigrator is initialized using a class imported from another package named knex-migrator:

const KnexMigrator = require('knex-migrator');
...
this.knexMigrator = new KnexMigrator({
  knexMigratorFilePath
});
...

Given that it originates from another package, it seems like a dead-end because it likely adheres to a standard protocol for database initialization that might be fundamentally incompatible with how PlanetScale is configured.

Additionally, tracing the code upward suggests there might be another package that needs forking to resolve this integration challenge between Ghost and PlanetScale.

Confronted with this challenge, I spent hours attempting to find a workaround to skip the database initialization within the ghost/core codebase. However, it turned out to be the REAL dead-end with the real breakthrough occurring when I decided to delve deeper into the "knex-migrator" package.

iii. Tracing knex-migrator package

Having exhausted other options, I decided to delve deeply into the "knex-migrator" package. The turning point in this ordeal occurred when I discovered that the "knex-migrator" package was owned by the TryGhost organization. You can view the repository here https://github.com/TryGhost/knex-migrator. This revelation indicated that the package was designed with Ghost in mind, and a potential solution might be just a simple pull request away from this repository.

Resuming the code tracing process, after investing some time, I successfully narrowed down the origin of the error when I identified the precise SQL command sent to PlanetScale. See line 126.

Notably, at line number 130, it became evident that the scenario where the database already exists had been accounted for. This discovery was somewhat surprising, considering the initial error reported by Ghost regarding the database already existing in PlanetScale. It implies that with PlanetScale, the SQL command triggers a different error number than the expected 1007 when attempting to create a database that already exists.

To confirm this, I inserted a console.error to log the error being caught at line 129, revealing:

Bingo! The error's errno is indeed different from 1007, which ultimately leads to the crash of the Ghost setup.

iv. Creating the patch

Based on the findings above, it appears that I don't necessarily need to create a Ghost setup option to skip database creation. Instead, I need Ghost to recognize that the error number originating from PlanetScale's database is not 1007.

To really resolve this issue, ensuring compatibility in error numbers is crucial. At this point, I don't have insight into where this difference in error numbers originates. It could be from the driver Ghost has utilized for communication with the MySQL database, or it might be specific to PlanetScale's implementation of the error. Individuals from the PlanetScale or Ghost's team reading this blog might have more ideas on this aspect than I do. 😀

But for my specific case, my primary goal is to achieve a functional deployment of Ghost. As a result, I am currently content with patching the currents/node_modules/knex-migrator/lib/database.js file with the following code:

  .catch(function (err) {
      // CASE: DB exists
      if (err.errno === 1007) {
          return Promise.resolve();
      }

      // Here's the patch
      const isPlanetScaleDBExists =
        err.errno == 1105 && 
        err.sqlMessage.endsWith("database exists");

      if (isPlanetScaleDBExists) {
          return Promise.resolve();
      }



      throw new errors.DatabaseError({
          message: err.message,
          err: err,
          code: 'DATABASE_CREATION_FAILED'
      });
}

To facilitate this, I crafted a script that patches Ghost during Docker build time.

file_path="current/node_modules/knex-migrator/lib/database.js"
text_to_add="const isDBExists = err.errno == 1105 && err.sqlMessage.endsWith('database exists');\\
if(isDBExists) return Promise.resolve();\\
"
line_number=129

sed -i "${line_number}i${text_to_add}" "$file_path"

The script above is designed to inject JavaScript code into the source code of the knex-migrator.

v. Final result

After crafting the patch, it took an additional hour or two to ensure my environment variables were correctly configured. Once everything fell into place, I was thrilled to finally deploy my Ghost website!

With the successful import of my blogs from the previous deployment, this Ghost site was up and running smoothly, with the database now hosted on PlanetScale! 🙌

Final Thoughts

Thank you for reading!

I appreciate you taking the time to read up to this point ❤️. If you find any parts confusing or have any issues with replication, I'd be happy to help. Just shoot me an email (thepiesaresquared@gmail.com) or DM/tweet me at @justfizzbuzz.

I thoroughly enjoyed the process of making PlanetScale and Ghost work seamlessly together! If you're interested in more posts like this, I invite you to subscribe to my blog, or let's connect and share our posts on Twitter!

Thank you 👨‍💻

CRDTs: A beginner's overview for building a collaborative app

jxtro — Sat, 03 Dec 2022 11:50:26 +0000

This is also published in my blog

Conflict-free replicated data types (CRDT) is one data structure I found truly fascinating. Basically, it enables distributed machines to eventually sync their data over time, i.e., eventual consistency.

To illustrate what I mean, consider the following scenario:

Two person, A and B, are doing collaborative work in a shared document, e.g., Google Docs
There is no centralized server managing the state b/w A and B.
A sends state updates to B, then B changes it's state to reflect A's updates, and vice versa.

Now, for some unknown reason, they got disconnected from each other for some time intervals. During this time, both person made changes to the document. Once they are reconnected, how should the program resolve their updates such that both have equivalent states?

One simple approach is to append timestamps to the updates and order events based on the timestamps. This approach, however, is predicated on the assumption that both machines have synchronous clocks that are immune to drifting and modification. This may result in data corruption depending on your use case.

CRDTs are intended to address this issue. Essentially, the goal is to structure the state updates in such a way that merge conflicts cannot occur*. This is accomplished by creating a data structure that has the following properties:

Merging updates are commutative - which means that if we have a MERGE function that applies state updates, the order in which each update is applied should be irrelevant to the resulting state.

C = MERGE( A, B) = MERGE(B, A)
Merging updates are idempotent - which means that merging the same updates more than once results in the same state.

C = MERGE( A, B ) = MERGE(C , A) = MERGE(C, B)

This is essentially combining updates in any order and frequency while producing the same result. This makes it ideal for distributed systems where communication packets may arrive duplicated, late, or in an erroneous order.

This approach is quite powerful and is widely used in distributed systems today. It's the ability to resolve conflicts in distributed databases as well as synchronize multiple nodes of a distributed system makes it an essential tool in today's systems.

Playing with CRDTs

There are a lot of implementations of CRDTs out there. In JavaScript, for instance, we have Y.js (https://github.com/yjs/yjs) and automerge (https://github.com/automerge/automerge). There’s also a Y.js demo (https://demos.yjs.dev/prosemirror/prosemirror.html) that allows you to play around with them and have your collaborative app running in just a few seconds. All messages are exchanged via webRTC while the state is managed via CRDTs. This can be a great sandbox to understand how CRDTs work.

I also attempted to build an offline-first application with Y.js-powered device syncing. The goal is to create a habit-tracking app that does not rely on a central server (here).

@justfizzbuzz

I had a lot of fun messing around with it. Overall, CRDTs are an incredibly useful structure for distributed systems and are gaining a lot of momentum. It's definitely worth investing time in them to better understand how they work and how you can apply them.

Conclusion

In conclusion, CRDTs provide an efficient way to keep distributed systems in sync. By structuring their data in such a way that merging is commutative and idempotent, they are able to maintain the same consistency regardless of delivery order and duplicate packets. This technique is widely used in distributed systems today and is gaining more and more traction. It’s definitely worth looking into this structure and understanding how it works and how you can apply it in your own systems.

* I may be relaxing some jargon here, what I mean is that there’s a deterministic algorithm or scheme that is guaranteed to combine any state updates such that no merge conflict, that demands manual resolution, is produced.

References

DEV Community: jxtro

Kubernetes vs Philippine Power Outages - On setting up k0s over Tailscale

I. The Challenge

Side-note: Why not host on cloud?

Back to the challenge

Kubernetes setup

Instructions

1. Manual node setup

2. Connect them over VPN

3. Install k0s

4. Configuration

Tailscale-connected Nodes

Private Container Registry

Networking

Node-local load balancing

Use your Kubernetes cluster

Final Thoughts

Self-hosting Ghost with Docker and PlanetScale

Part 1 - Setting up Ghost and PlanetScale

Some warning

A. Setting up database

1. Create a database in PlanetScale

2. Activate "Foreign key constraints" in your database.

B. Setting up Ghost

1. Install Ghost.

2. Customize your Ghost installation.

3. Patch the Ghost installation.

C. Connecting to the database

D. Run your Ghost instance

E. Deployment (Optional)

F. Conclusion

Part 2 - Integrating Ghost 🤝 PlanetScale

A. Configuring the value for database.connection.ssl

B. Foreign key constraints support for PlanetScale.

C. Addressing Ghost setup failures during initialization

i. The Problem

ii. Tracing Ghost's code

iii. Tracing knex-migrator package

iv. Creating the patch

v. Final result

Final Thoughts

CRDTs: A beginner's overview for building a collaborative app

Playing with CRDTs

Conclusion

A. Configuring the value for `database.connection.ssl`