DEV Community: Eduard

How to host your own radicle node. Contribute to decentralized source control. 🌌

Eduard — Sat, 17 Aug 2024 17:54:19 +0000

The other day there was a massive GitHub outage.
This made me realize how much we have come to depend on a single company to host not only our code, but our collaboration itself.

I have heard about radicle before, so I took this opportunity to delve more into what it is, and how it actually works.

Radicle is an open source, peer-to-peer code collaboration stack built on Git. Unlike centralized code hosting platforms, there is no single entity controlling the network. Repositories are replicated across peers in a decentralized manner, and users are in full control of their data and workflow.

I encourage you right now to read their introduction and then come back to continue, it's a really interesting project.

Radicle depends on its users to seed the repositories, much like torrenting does.
The community is still young, so I decided it would be great to support it on its early days to help it grow faster.

To seed is to give back. By seeding repositories on the Radicle network, you offer bandwidth, storage, and data availability to Radicle users.

If you are interested in supporting radicle like me, you can follow the following guide.

setting up a radicle seed node

I decided to use a cheap Hetzner machine. But you can use any existing server you have or any provider you prefer. Once you have your server ready, just ssh into it. You will need root access.

Note: You will need a domain name so that we can configure a DNS record to point to our node.

setup

We first need to create the user that will run the node. All data will be kept in their home directory.

# as root
groupadd --system seed
useradd --system --gid seed --create-home seed

You'll need to install the CLI and the web UI from their downloads page. Get the ones for your
architecture:

curl -O -L https://files.radicle.xyz/releases/latest/radicle-1.0.0-rc.14-x86_64-unknown-linux-musl.tar.xz
curl -O -L https://files.radicle.xyz/releases/latest/radicle-1.0.0-rc.14-x86_64-unknown-linux-musl.tar.xz.sig
curl -O -L https://files.radicle.xyz/releases/radicle-httpd/latest/radicle-httpd-0.15.0-x86_64-unknown-linux-musl.tar.xz
curl -O -L https://files.radicle.xyz/releases/radicle-httpd/latest/radicle-httpd-0.15.0-x86_64-unknown-linux-musl.tar.xz.sig

Then verify the signature of the downloads:

ssh-keygen -Y check-novalidate -n file -s radicle-1.0.0-rc.14-x86_64-unknown-linux-musl.tar.xz.sig < radicle-1.0.0-rc.14-x86_64-unknown-linux-musl.tar.xz
# Good "file" signature with ED25519 key SHA256:iTDjRHSIaoL8dpHbQ0mv+y0IQqPufGl2hQwk4TbXFlw

ssh-keygen -Y check-novalidate -n file -s radicle-httpd-0.15.0-x86_64-unknown-linux-musl.tar.xz.sig < radicle-httpd-0.15.0-x86_64-unknown-linux-musl.tar.xz
# Good "file" signature with ED25519 key SHA256:mqjWN1YrPRDTcVTxB4IZPHyH+vXpjWSogi+3zezZ/rQ

Finally just install the components, making sure to include the new binaries in your PATH:

# as root
tar -xvJf radicle-1.0.0-rc.14-x86_64-unknown-linux-musl.tar.xz --strip-components=1 -C /usr/local
tar -xvJf radicle-httpd-0.15.0-x86_64-unknown-linux-musl.tar.xz --strip-components=1 -C /usr/local

Now switch to the new user to continue, we will create a radicle profile:

# as root
su - seed

# as seed user
rad auth --alias < YOUR NODE DOMAIN HERE e.g. rad.seeder.example >
# just press enter on the passphrase prompt, it's not necesary for our server deployment

configuration

Next edit your node config.json in your seed home directory. We will configure our node to have a public policy and we will tell it which address to advertize the node as.

// /home/seed/.radicle/config.json
{
  // ...
  "node": {
    "alias": "<YOUR DOMAIN HERE>",
    // ...
    "externalAddresses": ["<YOUR DOMAIN HERE>:8776"], // <- update this
    // ...
    "seedingPolicy": {
      "default": "allow", // <- update this
      "scope": "all"
    }
  }
  // ...
}

To learn more about your node's policy, checkout the seeder guide section on policies.

Lets run the node to make sure it works:

# as seed user
rad node start --foreground
# you should see lots of output telling us about connected peers, fetched resources, etc...
# press C-c to exit

Lets create a system service unit so that we can configure our node to start with the server. Radicle provides a minimal systemd service unit file here.
We can download it and install it on our system:

# as root
curl -sS https://seed.radicle.xyz/raw/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/570a7eb141b6ba001713c46345d79b6fead1ca15/systemd/radicle-node.service -o /etc/systemd/system/radicle-node.service

If you followed the guide as is, you should not have to modify the service unit configuration.
Otherwise adjust as needed. Finally just enable the service:

# as root
systemctl enable --now radicle-node

# make sure service is working correctly
systemctl status radicle-node

# as seed user
rad node status

Optionally you can disable login as the seed user which is recommended:

# as root
chsh -s /usr/sbin/nologin seed

Congrats! At this point your node will seed public repositories! If you followed with me until now, thank you for contributing!

web ui

If you want to be able to see your node in web frontends such as this, you can do the following:

# as root

# enable the http api
curl -sS https://seed.radicle.xyz/raw/rad:z3gqcJUoA1n9HaHKufZs5FCSGazv5/570a7eb141b6ba001713c46345d79b6fead1ca15/systemd/radicle-httpd.service -o /etc/systemd/system/radicle-httpd.service
systemctl enable --now radicle-httpd

# you will need https support to appear in the web frontend, caddy is easy to configure
apt-get install caddy
curl https://raw.githubusercontent.com/caddyserver/dist/master/init/caddy.service -o /etc/systemd/system/caddy.service

# edit /etc/caddy/Caddyfile
<YOUR DOMAIN HERE> {
    reverse_proxy 127.0.0.1:8080
}

systemctl enable --now caddy

At this point you should be able to visit https://app.radicle.xyz/nodes/
and check your node!

Notes:

Remember to configure your DNS records to point your domain to your node IP address.
Remember to configure any firewalls to allow UDP:8776 and TCP:443

How to make easy encrypted backups with rclone for free

Eduard — Thu, 15 Aug 2024 17:11:38 +0000

Rclone supports tons of different storage backends.
I like to use R2 on Cloudflare with their generous free tier, but you can use AWS S3, Google Cloud Storage,
Dropbox, or even FTP among others.

You will need to install rclone first:

# ubuntu
sudo apt install nmap
# arch
sudo pacman -S rclone

If you don't have your storage bucket ready, create a cloudflare account then go into the R2 section and create a new bucket.

You will also need a new R2 token.

Make sure to give it access to your bucket.

Use rclone config to configure your storage option.
Follow this instructions for R2. Find all
supported backends here if you want to use a different storage option (like your personal FTP server).

My rclone.conf looks like this after setting up R2:

[r2]
type = s3
provider = Cloudflare
access_key_id = REDACTED
secret_access_key = REDACTED
region = auto
endpoint = https://REDACTED.r2.cloudflarestorage.com

At this point we should be able to use rclone to copy files to R2. We
probably don't want our files to end up in a bucket in plain text, so lets add some encryption.

Rclone supports a couple of useful "virtual" or "wrapper" storage remotes. We
can use the crypt remote to transparently encrypt
and decrypt operations with another remote.

Drop into rclone config one more time and
follow the example configuration.
When specifying the remote, chose your previous remote and specify a bucket
and optionally a base directory for your backups. I like to use r2:archive/backups;
archive is the bucket and backups/ will be the prefix for all operations.

I chose to disable filename encryption since we will be uploading archive files and
not replicating the filesystem itself. Chose a strong password you will remember.

Now my config looks like this:

... r2 entry ...

[backup]
type = crypt
remote = r2:archive/backup
filename_encryption = off
password = REDACTED

At this point we can use rclone to upload encrypted files to your backup bucket.
Lets make some utility functions to make this easier.

I like to use the fish shell, so this is what I include in my fish config to make my
backups:

function generate_backup
    set _name "$argv[1]".tar.zst

    sudo tar \
      --create \
      --absolute-names \
      --one-file-system \
      --preserve-permissions \
      --exclude-vcs \
      --exclude-caches \
      --exclude-backups \
      --exclude-tag-all=.NOBACKUP \
      --warning=no-file-ignore \
      --sort=inode \
      --zstd \
      --file=- \
        $argv[2..-1] | \
    rclone rcat --quiet \
      backup:"$_name"

    echo completed backup $_name
end

This function will archive and zip the given files, excluding some unwanted files,
then stream it into rclone, which will encrypt and upload to our backups bucket.
The first argument will be the backup name, which will be suffixed by a timestamp.

The --exclude-tag-all=.NOBACKUP line allows you to drop a .NOBACKUP file in
any directory and the backup will ignore that whole directory.

Now you can use this function whenever you want to make a backup, for example:

# creates backup/home-(timestamp).tar.zst.bin in your bucket
# includes all contents of desktop documents and videos
generate_backup home ~/Desktop ~/Documents ~/Videos

Making a real-time chatroom app with cloudflare workers

Eduard — Sat, 02 Dec 2023 20:46:30 +0000

Serverless Chatroom

In this tutorial, we will build a basic chatroom using Hono, Cloudflare Workers, HTMX, and Durable Objects.

For the full working example checkout the github repository.

Introduction

If you are not familiar with Cloudflare Workers, it is a platform that allows us to build serverless applications that are globally available, while being really cheap and easy to deploy.

Cloudflare Workers also allow us to use websockets in a serverless fashion. https://github.com/cloudflare/workers-chat-demo/blob/master/src/chat.mjs that showcases this features, but it is somewhat outdated.

In this post I want to show how to make a modern Cloudflare Worker that uses WebSockets and Durable Objects for a simple Chatroom web app.

Notes: You don't need a Cloudflare account to follow the tutorial locally, but if you want to deploy the project you will need a Paid plan to access the Durable Objects bindings.

Getting Started

We will use Hono as our web framework, Hono is a modern alternative to frameworks like Express or Koa that supports Cloudflare Workers directly.

$ npm create hono my-app
# make sure to select Cloudflare Worker as template
$ cd my-app && npm install

Our app now looks like this:

import { Hono } from 'hono'

const app = new Hono()

app.get('/', (c) => c.text('Hello Hono!'))

export default app

we can add endpoints just like with Express, and c is the request context that includes everything from Request parameters to response helpers such as the c.text method.

Use npm run dev to start the app and visit the link in your browser, you should see a response.

At this point you could also do npm run deploy to deploy the app to Cloudflare, you will have to authenticate with your Cloudflare account.

Making a basic frontend

For the actual web site, we will use HTMX with the WebSocket extension. This will allow us to extremely easy connect to our backend using a WebSocket, and sending messages and rendering new content in real time.

We can use the c.html helper from Hono to render our HTML:

import { html } from "hono/html";
// ...
app.get("/", (c) => {
    return c.html(html`
      <!doctype html>
      <html lang="en">
        <head>
          <title>chatroom</title>

          <script
            src="https://unpkg.com/htmx.org@1.9.9"
            integrity="sha384-QFjmbokDn2DjBjq+fM+8LUIVrAgqcNW2s0PjAxHETgRn9l4fvX31ZxDxvwQnyMOX"
            crossorigin="anonymous"
          ></script>
          <script src="https://unpkg.com/htmx.org/dist/ext/ws.js"></script>
        </head>
        <body>
          <main hx-ext="ws" ws-connect="/connect">
            <h1>chatroom</h1>
            <ul id="messages"></ul>
            <form ws-send>
              <input type="text" name="message" />
              <button>send</button>
            </form>
          </main>
        </body>
      </html>
    `);
  });

This is what's going on in the html:

We are including htmx library together with WebSocket extension in the document head.
In <main>, we are using the htmx attributes hx-ext and ws-connect to tell htmx to connect to a WebSocket endpoint at /connect
The <ul> will be used to render received messages, more on that later.
The <form> uses the ws-send attribute to tell htmx that submissions of this form, should be sent as messages inside the WebSocket connection from before.

You can check the full documentation for the WebSocket extension.

And that's it! With this little html the frontend can be fully functional, now let's go into the backend.

Understanding Durable Objects and WebSockets in Workers

We still need to define that /connect endpoint in the server, so first, lets understand how Cloudflare Workers allow us to handle WebSocket connections.

A Durable Object is an instance of a javascript class that we can define, that will persist and allow us to coordinate state between multiple requests, or in our case, multiple WebSocket clients.

Durable Objects provide a set of WebSocket APIs that make it possible to answer to incoming messages in a serverless way, so that connections that don't send messages can stay open without incurring any charges at all.

The basic architecture of our Durable Object will be the following:

Accept new WebSocket connections.
Receive all incoming messages, and echo the message to all connected clients.

Creating a Durable Object

To define our object, all we need to do is export a class from our Worker, and specify it in our wrangler.toml worker configuration file:

export class Chatroom {
  constructor (state: any, env: any) {}
}

# add this to your wrangler.toml
[[durable_objects.bindings]]
# This is the name of the binding that will be availble in our worker
name = "CHATROOM"
class_name = "Chatroom"

[[migrations]]
tag = "v1"
new_classes = ["Chatroom"]

to understand more about this configuration options, you can check the Durable Object's documentation.

Accepting WebSocket connections

Now for the good part. We will define a new /connect endpoint that will proxy the request to our Durable Object. The Durable Object will then accept the WebSocket connection.

import { HTTPException } from "hono/http-exception";
// ... adding the endpoint
app.get("/connect", async (c) => {
  if (c.req.header("upgrade") !== "websocket") {
    throw new HTTPException(402);
  }
  // Notice how this name <CHATROOM> matches the binding name in wrangler.toml
  const id = c.env.CHATROOM.idFromName("0");
 const chatroom = c.env.CHATROOM.get(id);
  return await chatroom.fetch(c.req.raw);
});

so what is this doing?

We access the binding through the env property that Hono provides us.
We need to use an ID to tell the Durable Object which instance of the Chatroom we want to use. In this example all connections will use chatroom 0 but we could add private rooms functionality as well.
Finally, we use the chatroom.fetch API to talk to the Durable Object. Objects expose functionality using the fetch API just like our worker does.

Finally, we just need to define the functionality of the Durable Object, so lets make some modifications to the Chatroom class:

export class Chatroom {
  state: any;

  constructor(state: any, env: any) {
    /// The state object contains all of the Durable Object APIs 
    this.state = state;
  }

  // This is the main 'entry point' of our object
  async fetch(request: Request) {
    const pair = new WebSocketPair();
    this.state.acceptWebSocket(pair[1]);
    return new Response(null, { status: 101, webSocket: pair[0] });
  }

  /* WEBSOCKET EVENT HANDLERS */

  async webSocketMessage(ws: WebSocket, data: string) {
    const { message } = JSON.parse(data);
    this.state.getWebSockets().forEach((ws: WebSocket) => {
      ws.send(
        html` <ul id="messages" hx-swap-oob="beforeend">
          <li>${message}</li>
        </ul>` as string,
      );
    });
  }


  async webSocketClose(
    ws: WebSocket,
    code: number,
    reason: string,
    wasClean: boolean,
  ) {
    console.log("CLOSED", { ws, code, reason, wasClean });
  }

  async webSocketError(ws: WebSocket, error: any) {
    console.error("ERROR", error);
  }
}

now lets explain how this works:

The constructor receives a state object. This object holds all of the APIs Durable Objects can use.
The fetch method is what will receive and process the request we send in the /connect endpoint.

In the fetch handler, we are creating a new WebSocket pair and accepting the connection using the Durable Object API for WebSockets. This is what enables us to handler connections with event handlers in a serverless fashion.

We can then define three handlers: webSocketMessage, webSocketClose and webSocketError.

For this app, we only focus on the message handler. All we need to do is parse the message field that we are submitting, and then we can get a list of all connected clients and send the new message to them.

<ul id="messages" hx-swap-oob="beforeend">
  <li>${message}</li>
</ul>

the htmx WebSocket extension then processes the HTML response and uses out-of-band swaps to append the message to the list of messages.

Try opening the app in multiple tabs and send messages from them. You should see all messages appear in every tab.

Going further

You may notice that an unused connection might close automatically around ~2 minutes or so. Durable Objects can also set a socket auto response to enable a heartbeat that does not incurr charges.

Just add this to the Object class:

// somewhere inside the constructor
this.state.setWebSocketAutoResponse(
  new WebSocketRequestResponsePair("ping", "pong"),
);

And then update the client as well:

          <script
            src="https://unpkg.com/htmx.org@1.9.9"
            integrity="sha384-QFjmbokDn2DjBjq+fM+8LUIVrAgqcNW2s0PjAxHETgRn9l4fvX31ZxDxvwQnyMOX"
            crossorigin="anonymous"
          ></script>
          <script>
            htmx.createWebSocket = function (src) {
              const ws = new WebSocket(src);
              setInterval(function () {
                if (ws.readyState === 1) {
                  ws.send("ping");
                }
              }, 20000);
              return ws;
            };
          </script>
          <script src="https://unpkg.com/htmx.org/dist/ext/ws.js"></script>

This will make sure all clients send a ping message every 20 seconds, to which the server will reply with pong to keep the connection alive.

Conclusions

And that's everything! We built a serverless application that allows user to send and receive messages in real time using Durable Objects with WebSockets!

You can try adding more features, like private chatrooms, user identifiers, etc. You will notice that a new user won't see old messages, so you can also experiment with storing messages using a datastore like Cloudflare D1.

Be creative!

Understanding bevy's default plugins

Eduard — Wed, 13 Apr 2022 22:19:02 +0000

Liquid syntax error: 'raw' tag was never closed